Deleting retries history during KCP remediation

[castaval]

What steps did you take and what happened?

At a random moment, during the KCP remediation process, a machine is created without the history of previous retries. The remedation-in-progress annotation hangs on the KCP resource and is deleted after some time.

It turns out that the retry history is not saved.

What did you expect to happen?

Retry history is saved

Cluster API version

1.7

Kubernetes version

1.27.14

Anything else you would like to add?

I think that this happens when the resource is patched, and at the next stage of reconciliation the control plane is listed without these annotations.

Label(s) to be applied

/kind bug
/area control-plane

[fabriziopandini]

Could you kindly provide some more info about this issue/instructions about how to reproduce?

[castaval]

This happens with a large number of KCP resources. The problem did not appear during local setup. What I do:

Create KCP
Break the first machine
It starts recreating

Debugging showed that this happens because during the get request, a resource comes without the necessary annotations. Therefore, the machine is created without annotations. And during the next reconciliation, the retry annotation is removed as stale.