Consider removing userdata (startup) script from VM when the machine has been bootstrapped successfully

[oliver-goetz]

How to categorize this issue?

/area robustness
/kind enhancement
/priority 3

What would you like to be added:
The userdata script added to VMs by MCM usually contains the operating system config of type provision and some OS extensions specific enhancements. This content is meant to be used once when the VM is bootstrapped.
Thus, removing the script from the VM when the machine has been bootstrapped successfully avoids issues which might occur when the script runs multiple times.

Why is this needed:
The userdata script turned out to be a startup script at many cloud providers which runs every time the VM is booting. Since the script is only created when the VM is bootstrapped its content is outdated pretty soon.
When the VM restarts after a while the userdata script might overwrite parts of the config with outdated data.

While we created certain measurements to avoid such issues (see gardener/gardener#11208) there is still a risk of corrupted configs, e.g. VMs created before the mentioned fix has been applied are still writing outdated config data to the disk when they are rebooted.

At the moment we could still get rid of those machines by rolling them. However, this won't be possible for machines where in-place updates are enabled.

[unmarshall]

Hello @oliver-goetz,

We discussed this within the MCM team and we have a few questions.

1. User data script is meant to be used once when initializing the VM to make it into a k8s Node. The script therefore should be idempotent to restarts of the VM. Do you see an issue in making the script idempotent? I see that you have already raised Create a wrapper for OperatingSystemConfig provisioning bash script to ensure that it is executed only once gardener#11208 to ensure that there are no side effects if the script is invoked again. Is that not sufficient?
2. MCM does not know anything about what is contained in User data (see mcm-aws-provider). When creating a VM it sets the UserData field of RunInstancesInput (in case of aws provider). Similarly relevant field is set for other providers as well. Ideally from its current responsibility set it should not know about what is contained in UserData and should also not manage the lifecycle of user data. Can you clarify the reasons that you think that MCM should now remove this script upon successful bootstrapping?

From our perspective the responsibility for identifying that bootstrapping has been successfully completed, followed by subsequent removal of the init script should be the responsibility of GNA or any other component running on the VM which keeps tap on the bootstrapping process.

If you wish to discuss this further, then we are happy to schedule a meeting to learn more from you in case we have gaps in our understanding.

[oliver-goetz]

I lost track of this issue 😅

Yes, there is gardener/gardener#11208 which prevents the provisioning OSC from being executed multiple times. While this fixes most of the issues we had, there are still a couple of drawbacks.
Most importantly, the provisioning OSC is envisioned to be applied only when the node is bootstrapped. It should not be applied later, because this would tamper the configuration of the node. Afaik we were not fully aware that the userdata script runs every time the node is booting (this might be even different in the different landscapes) until we were facing some issues with gardener-node-agent .

These are the drawback I still see.

• The wrapper is a utility function only, so OS extensions have to include it into their implementation.
• There could be more bugs in the bootstrap script in the future (introduced in Gardener and its OS extensions)
• VMs could life for a very long time (the fix has been implemented in January 2025 and we still find nodes which run an older version of the userdata script). When there are more nodes using in-place updates the number of long living nodes and their lifetime will increase further.

So why should this be implemented in MCM?

• MCM is the only component which is touching VMs. GNA does not have access to the hyperscaler API to update its underlying VM and we do not plan to grant such permissions to it.
• The userdata script is somehow injected into the startup process of a VM, so GNA cannot disable it on OS level.

Let's discuss this in a meeting 🙂