Rollout updates to cluster config the same as for upgrades

[Xartos]

What would you like to be added (User Story)?

As an operator I would like a cluster using topology to do rolling updates one machine deployment at a time when I update configuration of the cluster.

Detailed Description

When operating a cluster that is setup using the clusterclass/topology feature it very nicely updates one machine deployment at a time when doing kubernetes upgrades.
However when changing configuration of the nodes it will do a rollout of every machine deployment in parallel.

It would be nice if you could configure the topology controller to have the same rollout method for configuration changes as with kubernetes upgrades.

Anything else you would like to add?

It would also be very nice if you could add some more complex strategies. Like that you can tell the controller to do some machine deployments in serial and some in parallel.
As an example: If you have 3 machine deployments (in separate zones) for a db and 3 for applications. All the db nodes should be done in serial and all the application nodes in serial. But the groups could be done in parallel.

But this feels like a streched goal, maybe it merits its own feature request issue?

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[chrischdi]

We have the following two annotations:

topology.cluster.x-k8s.io/hold-upgrade-sequence

• It can be used to hold the entire MachineDeployment upgrade sequence. If the annotation is set on a MachineDeployment topology in Cluster.spec.topology.workers, the Kubernetes upgrade for this MachineDeployment topology and all subsequent ones is deferred.

topology.cluster.x-k8s.io/upgrade-concurrency

• It can be used to configure the maximum concurrency while upgrading MachineDeployments of a classy Cluster. It is set as a top level annotation on the Cluster object. The value should be >= 1. If unspecified the upgrade concurrency will default to 1.

Did you consider if these fit your requirements?

Link to the docs: https://main.cluster-api.sigs.k8s.io/reference/api/labels-and-annotations.html?highlight=annotations#supported-annotations

[Xartos]

But those annotations are only for the upgrade procedure right? i.e. when you upgrade kubernetes. Because I think that that flow works perfectly.

The issue I'm having is if I change anything else on the kubeadmconfigtemplates, let's say I'm changing SSH-keys for the nodes. Then I'd like the same behavior as for when upgrading kubernetes. And not need to manually pause each machine deployment and unpause them one by one.

[chrischdi]

Yes, I think you're right, its only used when actually upgrading, not for other fields 🤔

[chrischdi]

Note for:

It would also be very nice if you could add some more complex strategies. Like that you can tell the controller to do some machine deployments in serial and some in parallel.
As an example: If you have 3 machine deployments (in separate zones) for a db and 3 for applications. All the db nodes should be done in serial and all the application nodes in serial. But the groups could be done in parallel.

This can already be done today when using PodDisruptionBudgtes on the workload side.

[fabriziopandini]

My hot take:

• the idea of having some control on any update - and not only on upgrades - seems an interesting use case to look at.
• figuring out what this entails and a better UX - possibly not based on annotations - requires some research e.g.
  • we specifically added infos to various CR in order to track upgrade progress, I'm not sure we have a similar signal for any upgdate progress