Remove Yum Updates in Cloud-Init #1074
Conversation
bwagner5
force-pushed
the
opp-yum-updates
branch
from
64f64a8
to
29129d0
Compare
bwagner5
force-pushed
the
opp-yum-updates
branch
4 times, most recently
from
1be312e
to
dde694d
Compare
bwagner5
force-pushed
the
opp-yum-updates
branch
2 times, most recently
from
53cd26c
to
122940c
Compare
|
I buy the version skew argument. In your second test, when updates were installed; what packages were affected? I've never really understood what gets classified as "security-related" for this behavior. The kernel obviously couldn't be updated because that would require a reboot. What would we be giving up by removing this? |
The second test was a systemd update that happened in the wild with the previous AMI release. The packages were updated but didn't take affect since it would have required a reboot anyways. So the 8 seconds was just checking and then downloading the packages that weren't even installed. |
|
To answer your question more directly on what we would lose out on: Some of packages that can be updated would be a CVE patch for curl or sshd. In curl's case, the binary would be updated and the next invocation of it would be patched. In sshd's case, the binary would be updated but the service wouldn't be restarted unless the user restarted it manually within user-data. |
files/cloud.cfg
Outdated
| @@ -0,0 +1,82 @@ | |||
| # WARNING: Modifications to this file may be overridden by files in | |||
| # /etc/cloud/cloud.cfg.d | |||
There was a problem hiding this comment.
is there an upstream we can add a reference to here, so we can rebase on the defaults every once in a while? did you just grab this off an AL2 instance?
seems like this might be the source of truth, but it's a jinja template 🤢 https://github.com/canonical/cloud-init/blob/main/config/cloud.cfg.tmpl
There was a problem hiding this comment.
Yeah I grabbed it off an instance. That template resolved isn't what is shipped in AL2. I asked and it looks like it is only available in the cloud-init RPM in the amzn2 yum repo. It's a little unfortunate, but I'm also told it's rarely updated. The version of cloud-init on AL2 is pretty old too:
> cloud-init --version
/bin/cloud-init 19.3-46.amzn2
The latest release upstream is 22.3.4
There was a problem hiding this comment.
Ah bummer. I guess you already considered patching cloud.cfg instead of wholesale replacing it? Doesn't seem like there's a way we can disable this module with a /etc/cloud/cloud.cfg.d drop-in either
There was a problem hiding this comment.
Yeah, definitely went down that path first and it doesn't look like there's a way to do with with /etc/cloud/cloud.cfg.d.
There was a problem hiding this comment.
Changed it to just sed the file to remove that one module.
bwagner5
force-pushed
the
opp-yum-updates
branch
from
122940c
to
1547897
Compare
|
Does this PR break usage of meaning the package isn't installed... (reverting to the previous ami works) Is that the expected outcome? |
Issue #, if available:
#1099
Description of changes:
Generally, the cloud-init AL2 config checks for and updates some security related packages via the 'package-update-upgrade-install' module in the cloud-init-config stage (before user-data). This usually results in a 5 second delay executing user-data, even though updates are not required a lot of the time, if AMIs are being updated frequently. The main latency when checking for updates is the yum cache hydration in
/var/cache/yumwhich downloads ~700M of data when the first yum update is executed.It would be possible to include the cache in the AMI, but we would need to create an image per region which we currently do not do.
Additionally, updating packages on startup can result in version skew across a cluster of instances using the same AMI version. This is undesirable from a configuration perspective and could cause stability problems in the case of an update breaking a node's bootstrap process.
In the following cloud-init analyze table, notice that
config-package-update-upgrade-installtakes 5 seconds to complete when no updates are needed. If updates are required, this module can take anywhere from 5-10 seconds.No Updates are required:
Updates are required:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Testing Done
Created AMI, launched, observed going to ready
See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.