We need to get security updates onto instances on live AWS services. So, whats the best strategy? If we’re using the Amazon Linux AMI, then we security updates are automatically applied on the initial boot of the AMI. So if we cycle our instances, we get a freshly updated EC2 instance.
But, if we’re using an Ubuntu AMI, we need to do a little more work depending on what compromises we’re looking to make. Key to this decision is the balance between the need to get security updates quickly, and the time taken to reboot an instance to apply some core updates to components such as the kernel or glibc.
In summary, we have a two basic methods,
Upgrading packages on the current system
(i) Package update only - We update packages but don’t reboot. So we get *most* updates, but not core ones such as to the kernel. The benefit is we avoid costly rebooting and downtime. Ubuntu has a package ‘unattended-upgrades’ that we can set up to automatically take care of this for us.
(ii) Package update and reboot - We get all the updates as above *plus* core ones. The downside is we have to then endure a reboot, this can open up the chance that our auto-scaling group might terminate the instance before its live again, so we need to be extra careful with our scaling policy timings.
Upgrading packages on a new instance
(i) Create fresh instance and update with cloud-init - We can set up cloud-init to get all security packages on first boot, this means we get the same result as the the first option, all non-reboot security updates. An important benefit of this approach though is that we can maintain our current instances in our auto-scaling group while we create and update the new instance before scaling back, ensuring that we have no downtime.
(ii) Create fresh instance, update with cloud-init and reboot after upgrade - The same as the above but with a reboot of the fresh instance after upgrade.
For our best security, we’re going to combine the two methods, we could also bring in automated instance cycling to our AWS system as default.
-
Start up a new instance in our auto-scaling group
-
Cloud-init then applies all security upgrades and reboots
-
New instance is ‘InService’ in the auto-scaling group
-
Kill one of the old instances
-
New instance regularly applies security upgrades through the unattended-upgrades package
Pros:
-
We get all our security updates on a fresh instance and get non-reboot updates subsequently.
-
Instances are regularly cycled, making them more ephemeral and assuring us our services have no manual-magic holding them together.
Cons:
-
It takes more time to up an instance on autoscaling, now we need to upgrade and reboot.
-
New instances are not the same as the old, there might be inconsistencies or problems introduced by new package versions.
In the next section I’ll describe how to implement unattended upgrades using saltstack config management, and a way of automatically cycling our instances in a minimising downtime way.
Unattended updates using a salt state
https://github.com/saltstack-formulas/apt-formula/tree/master/apt
Using cloud init and instance cycling
- Use launch configuration user data to
Set cloud init to apt update and upograde, rebooting if required
