MachineSet controller scales up machines during cluster deletion after restart

[bfbachmann]

What steps did you take and what happened?

1. Create a cluster with any number of machines.
2. Once the cluster is ready, take down the pod(s) running CAPI controllers (I just set their image tag so some invalid tag and waited for them to go into ImagePullBackoff).
3. Bring the controller pods back up, and while they're booting, issue the command to delete the cluster until the request actually gets a successful response – usually this happens before all the controllers/workers/eventsources have been fully initialized. (Our system that uses CAPI does this – it retries the cluster delete command until it gets a successful response).
4. Wait for CAPI to delete the cluster.

You'll notice that CAPI actually creates new machines while deleting the existing ones in the cluster, and then it stalls for a while (maybe 15-20mins, but certainly far longer than usual) before it finally deletes the cluster.

What did you expect to happen?

The cluster is deleted after some reasonable period of time (just after the underlying infrastructure is deleted).

Cluster API version

v1.4.4

Kubernetes version

v1.26.9+k3s1

Anything else you would like to add?

Here's an annotated version of the CAPI logs from a case where this happened to me.

1. At this point, capi-controller-manager has just started. All the logs before this point are just CAPI saying that it's registering webhooks and starting workers and controllers.

2. Seems like CAPI just noticed that we're deleting a cluster.

18:18:36.495104       1 cluster_controller.go:241] "Cluster still has children - deleting them first" controller="cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="fleet-default/bruno1" namespace="fleet-default" name="bruno1" reconcileID=d3cb487a-4f72-49f3-a5d0-5ae9ada54284 count =1

3. CAPI deletes the ONLY MachineDeployment in this cluster

18:18:36.495116       1 cluster_controller.go:252] "Deleting child object" controller="cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="fleet-default/bruno1" namespace="fleet-default" reconcileID=d3cb487a-4f72-49f3-a5d0-5ae9ada54284 gvk="cluster.x-k8s.io/v1beta1, Kind=MachineDeployment" name="bruno1-pool1"

4. Clearly the capi-controller-manager hasn't fully started up yet because some controllers aren't running yet. Not sure if this matters.

18:18:36.495368       1 controller.go:228] "Starting workers" controller="machinedeployment" controllerGroup="cluster.x-k8s.io" controllerKind="MachineDeployment" worker count=10
18:18:36.495392       1 controller.go:228] "Starting workers" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" worker count=10
18:18:36.495832       1 controller.go:228] "Starting workers" controller="machinehealthcheck" controllerGroup="cluster.x-k8s.io" controllerKind="MachineHealthCheck" worker count=10

5. At this point there is still only one MachineDeployment and one MachineSet for the cluster. This is correct.

18:18:36.509157       1 cluster_controller.go:267] "Cluster still has descendants - need to requeue" controller="cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="fleet-default/bruno1" namespace="fleet-default" name="bruno1" reconcileID=d3cb487a-4f72-49f3-a5d0-5ae9ada54284 descendants="Machine deployments: bruno1-pool1;Machine sets: bruno1-pool1-598fbdc96dxb6jwb" indirect descendants count=1

6. At this point it seems like the MachineDeployment deletion has taken effect as we no longer see the it in this log (unlike in the one above). This seems fine so far.

18:18:36.522988       1 cluster_controller.go:267] "Cluster still has descendants - need to requeue" controller="cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="fleet-default/bruno1" namespace="fleet-default" name="bruno1" reconcileID=294849c4-d9da-4417-b6d0-92debb9b445c descendants="Machine sets: bruno1-pool1-598fbdc96dxb6jwb" indirect descendants count=1

7. CAPI is adding a watcher to the existing machine in the cluster.

18:18:36.601989       1 tracker.go:57] "Adding watcher on external object" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=02c2078a-3f71-4ac4-86cc-23119523605b MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" MachineDeployment="fleet-default/bruno1-pool1" Cluster="fleet-default/bruno1" groupVersionKind="rke.cattle.io/v1, Kind=RKEBootstrap"
18:18:36.602007       1 controller.go:151] "Starting EventSource" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" source="kind source: *unstructured.Unstructured"
18:18:36.611476       1 tracker.go:57] "Adding watcher on external object" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=02c2078a-3f71-4ac4-86cc-23119523605b MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" MachineDeployment="fleet-default/bruno1-pool1" Cluster="fleet-default/bruno1" groupVersionKind="rke-machine.cattle.io/v1, Kind=DigitaloceanMachine"
18:18:36.611537       1 controller.go:151] "Starting EventSource" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" source="kind source: *unstructured.Unstructured"

8. CAPI is refusing to delete the machine because the associated cluster has a non-zero deletion timestamp. This is expected, I think. See https://github.dev/kubernetes-sigs/cluster-api/blob/00dbf7b9f6322d7ebd06ae2efa703b23354dd37d/internal/controllers/machine/machine_controller.go#L318.

18:18:36.620957       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=468785aa-80f4-4c88-8428-32b1f98eb133 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"

9. CAPI's MachineDelpoyment controller is trying to reconcile the MachineDeployment "bruno1-pool1", but it can't because it already deleted it above. I think this is intended/normal.

18:18:36.641444       1 machinedeployment_controller.go:167] "Failed to reconcile MachineDeployment" err="failed to update revision annotation on MachineDeployment: MachineDeployment.cluster.x-k8s.io \"bruno1-pool1\" not found" controller="machinedeployment" controllerGroup="cluster.x-k8s.io" controllerKind="MachineDeployment" MachineDeployment="fleet-default/bruno1-pool1" namespace="fleet-default" name="bruno1-pool1" reconcileID=b4115e4e-a299-4e15-88fe-74a7698fc3fb Cluster="fleet-default/bruno1"
18:18:36.641731       1 controller.go:329] "Reconciler error" err="failed to update revision annotation on MachineDeployment: MachineDeployment.cluster.x-k8s.io \"bruno1-pool1\" not found" controller="machinedeployment" controllerGroup="cluster.x-k8s.io" controllerKind="MachineDeployment" MachineDeployment="fleet-default/bruno1-pool1" namespace="fleet-default" name="bruno1-pool1" reconcileID=b4115e4e-a299-4e15-88fe-74a7698fc3fb

10. Here's where bad things really start to happen. I think CAPI is trying to reconcile the MachineSet for the cluster and sees that, because it already deleted the only machine in the set, it's running 0 machines when it should be running 1. So it tries to create a new machine even though other controllers are trying to delete the cluster.

18:18:36.650528       1 machineset_controller.go:456] "MachineSet is scaling up to 1 replicas by creating 1 machines" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" MachineSet="fleet-default/bruno1-pool1-598fbdc96dxcl6q6" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6" reconcileID=79055edb-235b-457f-ba5d-34657880a4db MachineDeployment="fleet-default/bruno1-pool1" Cluster="fleet-default/bruno1" replicas=1 machineCount=0

11. The machine controller is waiting for the DigitaloceanMachine corresponding to the machine it deleted earlier to be deleted.

18:18:36.668726       1 machine_controller.go:413] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=468785aa-80f4-4c88-8428-32b1f98eb133 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-gpjrk"

12. Same error as before.

18:18:36.721011       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=92761182-3aa3-4f34-abed-c3a54c198b24 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"

13. The MachineSet controller just noticed that the new machine has been created successfully. By my logic, this machine (bruno1-pool1-598fbdc96dxcl6q6-hkbtj) should not have been created.

18:18:36.742123       1 machineset_controller.go:548] "Created machine 1 of 1" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" MachineSet="fleet-default/bruno1-pool1-598fbdc96dxcl6q6" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6" reconcileID=79055edb-235b-457f-ba5d-34657880a4db MachineDeployment="fleet-default/bruno1-pool1" Cluster="fleet-default/bruno1" RKEBootstrap="fleet-default/bruno1-bootstrap-template-stfgx" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-k6jjd" Machine="fleet-default/bruno1-pool1-598fbdc96dxcl6q6-hkbtj"

14. Same error as before.

18:18:36.810935       1 machine_controller.go:413] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=92761182-3aa3-4f34-abed-c3a54c198b24 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-gpjrk"
18:18:36.831737       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=34dae327-e65b-4fd9-bb98-a6e9f4fb7917 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"

15. The machine controller is erroring because the newly-created machine is not associated with any infra machine (DigitaloceanMachine). See https://github.dev/kubernetes-sigs/cluster-api/blob/00dbf7b9f6322d7ebd06ae2efa703b23354dd37d/internal/controllers/machine/machine_controller_phases.go#L101.

18:18:36.861993       1 machine_controller_phases.go:101] "could not find external ref, requeuing" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxcl6q6-hkbtj" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6-hkbtj"  reconcileID=89032bfa-b1c3-4551-a83d-4434d4fab5c6 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxcl6q6" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-k6jjd"
18:18:36.863625       1 controller.go:329] "Reconciler error" err="[MachineSet.cluster.x-k8s.io \"bruno1-pool1-598fbdc96dxcl6q6\" not found, machinesets.cluster.x-k8s.io \"bruno1-pool1-598fbdc96dxcl6q6\" not found]" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" MachineSet="fleet-default/bruno1-pool1-598fbdc96dxcl6q6" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6" reconcileID=79055edb-235b-457f-ba5d-34657880a4db

16. CAPI is still waiting for the infra machine (DigitaloceanMachine) for the original CAPI machine to be deleted.

18:18:36.897830       1 machine_controller.go:413] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=34dae327-e65b-4fd9-bb98-a6e9f4fb7917 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-gpjrk"

17. Same error as before.

18:18:36.899317       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxcl6q6-hkbtj" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6-hkbtj" reconcileID=33332c51-5ff8-4727-9a0c-c0e66d8d409c MachineSet="fleet-default/bruno1-pool1-598fbdc96dxcl6q6" Cluster="fleet-default/bruno1" Node="" cause="cluster is being deleted"
18:18:36.921383       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=841e8ca5-f316-4cff-aed6-bfeebf23bdc3 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"
18:18:36.946397       1 machine_controller.go:413] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=841e8ca5-f316-4cff-aed6-bfeebf23bdc3 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-gpjrk"
18:18:36.980253       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=faa124f0-86d1-4465-b2c5-8992307a87f5 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"

18. Still waiting.

18:18:37.023953       1 machine_controller.go:413] "Waiting for infrastructure to be deleted" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=faa124f0-86d1-4465-b2c5-8992307a87f5 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" DigitaloceanMachine="fleet-default/bruno1-pool1-af648dc8-gpjrk"

19. Same reconciler error.

18:18:37.033105       1 controller.go:329] "Reconciler error" err="machines.cluster.x-k8s.io \"bruno1-pool1-598fbdc96dxcl6q6-hkbtj\" not found" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxcl6q6-hkbtj" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxcl6q6-hkbtj" reconcileID=33332c51-5ff8-4727-9a0c-c0e66d8d409c
18:18:37.759150       1 controller.go:329] "Reconciler error" err="machinesets.cluster.x-k8s.io \"bruno1-pool1-598fbdc96dxb6jwb\" not found" controller="machineset" controllerGroup="cluster.x-k8s.io" controllerKind="MachineSet" MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb" reconcileID=507bde1a-7b8e-40c5-8264-00bdb4c42905
18:18:41.524210       1 cluster_controller.go:298] "Cluster still has descendants - need to requeue" controller="cluster" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="fleet-default/bruno1" namespace="fleet-default" name="bruno1" reconcileID=077874dd-e00a-47f2-b8d8-2f0e58162aca controlPlaneRef="bruno1"
18:18:51.926860       1 machine_controller.go:318] "Deleting Kubernetes Node associated with Machine is not allowed" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="fleet-default/bruno1-pool1-598fbdc96dxb6jwb-v94d2" namespace="fleet-default" name="bruno1-pool1-598fbdc96dxb6jwb-v94d2" reconcileID=c2c364d1-0775-4f7a-b079-1ce4060ecb17 MachineSet="fleet-default/bruno1-pool1-598fbdc96dxb6jwb" Cluster="fleet-default/bruno1" Node="bruno1-pool1-af648dc8-gpjrk" cause="cluster is being deleted"

Label(s) to be applied

/kind bug
/area machineset

[killianmuldoon]

Good catch!
If I'm understanding the case correctly the Cluster is eventually deleted so this isn't a deadlock, and this is an edge case that occurs when the controllers are restarted.

The way deletion should normally work is that the Cluster controller issues a delete command for the MachineDeployment and from there Kubernetes garbage collection should delete the objects based on the owner reference chain. The expectation is that a Machine will not be deleted until after the MachineSet is deleted, which is at odds with your experience here.

Two questions:

• What command are you running to delete the cluster?
• Does your cluster have foreground deletion enabled?

/triage accepted

[bfbachmann]

Thank you for the quick response, @killianmuldoon!

If I'm understanding the case correctly the Cluster is eventually deleted so this isn't a deadlock.

Correct. I have yet to see a case where the cluster never gets deleted. It's just slow to delete.

What command are you running to delete the cluster?

We're making an API call that is the equivalent of kubectl delete clusters.cluster.x-k8s.io <name>. The exact call is here if you'd like to see the code.

Does your cluster have foreground deletion enabled?

No, we're just using the default background deletion policy.

[killianmuldoon]

In this case then the question is how the Machine gets deleted in the first place. CAPI's deletion process would not delete the Machine before deleting the MachineSet as it relies on Kubernetes garbage collection for Machine deletion.

Is this issue reproducible in your setup? Can you see when and how the Machine is being deleted, and by what process?

[bfbachmann]

I think the Machine is being deleted after the MachineSet correctly. In other words, what I'm seeing is that the cluster controller logs Deleting child object when it's deleting the MachineDeployment, at which point the underlying MachinetSet and Machine get deleted properly as well according to their owner refs – I assume k8s is doing this as part of normal garbage collection. This part seems fine to me, it's just that the MachineSet controller creates a new machine immediately thereafter – that part is strange. This behaviour is perfectly consistent for me. It's also worth noting that the new machine that is create gets deleted almost immediately.

[killianmuldoon]

My understanding of the flow is that this should never happen - a Machine shouldn't be deleted until after the MachineSet is already fully deleted, so the MachineSet shouldn't be able to reconcile after the Machine is fully deleted. I don't think there can be a race condition here either - at least under normal conditions where a Machine takes some non-zero number of seconds to be deleted.

Could you add an arbitrary finalizer to the MachineSet preventing its deletion to see if something else is causing the Machine to be deleted?

[bfbachmann]

Could you add an arbitrary finalizer to the MachineSet preventing its deletion to see if something else is causing the Machine to be deleted?

When I do this, the following happens (in chronological order):

1. I set the finalizer on the MachineSet. And proceed with the reproduction steps above.
2. The finalizer I added to the MachineSet prevents both the MachineSet and the Machine from being deleted, as I believe it should. The MachineSet has its deletion timestamp set, but the corresponding Machine does not.
3. When I manually remove the finalizer from the MachineSet, the MachineSet is deleted, and shortly afterward the Machine has its deletion timestamp set, after which it is deleted.

It appears that the end result here is similar but not exactly the same. I don't see the additional machine getting created during cluster deletion like I did before, but the cluster still hangs in the Deleting state for a long time before being deleted.

[bfbachmann]

It's also worth noting that, for some reason, creating a new cluster seems to immediately cause any cluster stuck in Deleting this way to be removed.

[killianmuldoon]

Could you share an example MachineSet in full?

[bfbachmann]

apiVersion: v1
items:
- apiVersion: cluster.x-k8s.io/v1beta1
  kind: MachineSet
  metadata:
    annotations:
      machinedeployment.clusters.x-k8s.io/desired-replicas: "1"
      machinedeployment.clusters.x-k8s.io/max-replicas: "2"
      machinedeployment.clusters.x-k8s.io/revision: "1"
      objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4xTTW/bMAz9KwPPduasX6mBXfZxKvaBoNudlmhXMy0JEp02KPLfB7lNYadx2qPpR76nR75H6EhQoyCUj4DWOkExzsb06ap/pCSSLIJxC4UiTAvjPhoNJYSWcsV9FAqQzULdvaWQN5sWSvDBbUw0zhrbjCCbZfbhxlj9+es7p1nsCEqoQm/d8l3o6FGllpqJJNdUY88CuwxUoOG5t6ajKNh5KG3PnAFjRTyYMJoXoQQ2tn84QXqH8Q5KOKf6TBVV9enq7Lq4pqK6UloV15eXuDqji+qiWBXLJeoiiZi8J/fOcXrVSd3RkxrEPVn288CRQJ6NwgjlMoNITEpcgPIxdUpAoWabuoNjNrb54zUKpYImJqHfjo3aQgm/WFMc+GTrE8F60pDq1Hl+bh4f0pv2PQtfPOTtKqa/z4VXy30NdFaC49wz2oSU0NMxnCbPbtuRlfy4waGl0d7m+A9QY/I8OB4pmCJJlD4JSPnpUN0ZS4OkPfNxefcutBQmA3ejO6ick7RZP1jubG2aNdVDpL35SyGl7imzk+BBBq2xKc3rm+9f9jNu90s9PM0XllymkJk73WVzB2psHTBK6JX0gWak7u2ZkfzNNEaQnSK0P56Qs8oHU3OsL89XWq3eUD0YKyj9cMC4QcNYMa1fQlWkhKHeHlRGH7093tYPydGj2m73PwAA//80/evYgwUAAA
      objectset.rio.cattle.io/id: rke-cluster
      objectset.rio.cattle.io/owner-gvk: provisioning.cattle.io/v1, Kind=Cluster
      objectset.rio.cattle.io/owner-name: bruno1
      objectset.rio.cattle.io/owner-namespace: fleet-default
    creationTimestamp: "2023-11-01T17:49:19Z"
    generation: 1
    labels:
      cattle.io/os: linux
      cluster.x-k8s.io/cluster-name: bruno1
      cluster.x-k8s.io/control-plane: "true"
      cluster.x-k8s.io/deployment-name: bruno1-pool1
      machine-template-hash: 1549687528-rvnpm
      rke.cattle.io/cluster-name: bruno1
      rke.cattle.io/control-plane-role: "true"
      rke.cattle.io/etcd-role: "true"
      rke.cattle.io/rke-machine-pool-name: pool1
      rke.cattle.io/worker-role: "true"
    name: bruno1-pool1-598fbdc96dxjndgc
    namespace: fleet-default
    ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: MachineDeployment
      name: bruno1-pool1
      uid: 642cd460-cac6-4bac-baa4-b767a3df97de
    resourceVersion: "24980"
    uid: 783b813a-79d5-433e-8b09-99ddf6f2ee55
  spec:
    clusterName: bruno1
    deletePolicy: Oldest
    replicas: 1
    selector:
      matchLabels:
        cluster.x-k8s.io/cluster-name: bruno1
        cluster.x-k8s.io/deployment-name: bruno1-pool1
        machine-template-hash: 1549687528-rvnpm
    template:
      metadata:
        labels:
          cattle.io/os: linux
          cluster.x-k8s.io/cluster-name: bruno1
          cluster.x-k8s.io/control-plane: "true"
          cluster.x-k8s.io/deployment-name: bruno1-pool1
          machine-template-hash: 1549687528-rvnpm
          rke.cattle.io/cluster-name: bruno1
          rke.cattle.io/control-plane-role: "true"
          rke.cattle.io/etcd-role: "true"
          rke.cattle.io/rke-machine-pool-name: pool1
          rke.cattle.io/worker-role: "true"
      spec:
        bootstrap:
          configRef:
            apiVersion: rke.cattle.io/v1
            kind: RKEBootstrapTemplate
            name: bruno1-bootstrap-template
            namespace: fleet-default
        clusterName: bruno1
        infrastructureRef:
          apiVersion: rke-machine.cattle.io/v1
          kind: DigitaloceanMachineTemplate
          name: bruno1-pool1-af648dc8
          namespace: fleet-default
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: "2023-11-01T17:52:54Z"
      status: "True"
      type: Ready
    - lastTransitionTime: "2023-11-01T17:49:19Z"
      status: "True"
      type: MachinesCreated
    - lastTransitionTime: "2023-11-01T17:49:29Z"
      status: "True"
      type: MachinesReady
    - lastTransitionTime: "2023-11-01T17:52:54Z"
      status: "True"
      type: Resized
    fullyLabeledReplicas: 1
    observedGeneration: 1
    readyReplicas: 1
    replicas: 1
    selector: cluster.x-k8s.io/cluster-name=bruno1,cluster.x-k8s.io/deployment-name=bruno1-pool1,machine-template-hash=1549687528-rvnpm
kind: List
metadata:
  resourceVersion: ""

[killianmuldoon]

but the cluster still hangs in the Deleting state for a long time before being deleted

What's a long time in this case? Which objects are taking time to delete? This could be down to a lot of different factors - are you comparing with past behaviour or with other clusters?

Your findings with the finalizer suggest that there isn't another component deleting the Machine, and that foreground deletion isn't somehow at work. I'm really not sure what else could be the root cause for this.

Is there any chance you see the same behaviour with CAPD? I haven't been able to reproduce it myself.