Rolling my own exponential backoff

[mogsie]

I'm aware of the exponential backoff provided by kubebuilder: return an error or just set requeue to true. This causes controller-runtime to requeue with exponential backoff. What id I want the exponential backoff to start slower than just a few milliseconds? The reason is to avoid bing rate-limited by other APIs.

For example, in my reconcile, I might determine that object X failed to reconcile, and should be reconciled after ~1 second, and if it fails, retry, but exponentially backing off at ~2 seconds, then ~4 and so on. The same as what controller run-time does now, except skip the first second of exponentially backing off.

[mogsie]

I'm thinking something like a denyBefore map, a mapping from names to timestamps which would work like this.

• When I determine that this object shouldn't be reconciled for a second, and then exponentially back off, I add denyBefore["namespace/name"] = time.Now + 1 * time.Second — essentially: deny reconciling this object for 1 second
• When the reconcile loop starts, it immediately checks the denyBefore, and if the re is an entry for the object, immediately return requeue: true to get the "normal" exponential backoff.
• If instead the timestamp has passed, remove it from the denyBefore map if it was there and reconcile as normal.

This would have the following effect for these "Denied" objects:

• for ~1 second, it would be reconciled, but the reconciler will exit immediately, asking to be requeued.
• a short time after 1 second, the object would be reconciled as normal
• if that reconcile fails with an error, for example, then the backoff would already be at >1 second, and so it would start exponentially from ~1 second.

[camilamacedo86]

Hi @mogsie,

I understand that the error that you are facing is the object has been modified, please apply your changes to the latest version and try again which is indicative of a conflict due to concurrent updates on the same Kubernetes resource. When this happens, your reconciler will fail and, by default, will immediately requeue the request, potentially exacerbating the problem.

Implementing a denyBefore map, as you described, can help introduce an intentional delay before requeueing. This can give other concurrent operations a chance to complete and thus reduce the chances of subsequent conflicts.

The ideal approach to sort out this scenario is fetch the resource before update as it is implemented when you use the Deploy Image plugin to generate the scaffold, see the controller implementation under the testdata samples, such as here: https://github.com/kubernetes-sigs/kubebuilder/blob/689360500f8c218f69f2aa47aaa26da26b179e16/testdata/project-v4-with-deploy-image/internal/controller/busybox_controller.go#L109C3-L118

On top of that see that:

• Concurrency: Controller-runtime's reconciliation is designed to handle multiple concurrent requests. If your controller scales and starts reconciling multiple objects concurrently, you might run into race conditions with the denyBefore map. To handle this, you would need to synchronize access to the map using mutexes.
• Memory Growth: Depending on the number of objects being managed and the error rate, the denyBefore map could grow significantly. Ensure to periodically clear old timestamps or implement a cleanup mechanism to avoid potential memory issues.
• Metrics/Logging: To debug any future issues or monitor the effectiveness of this approach, consider adding metrics or logging around these custom delays. This will help in understanding the behavior of your reconciler in real-world scenarios.

Then, if you want change since it would might either consider proposal Controller-Runtime Enhancements and discuss that in their repository.

Therefore, I am closing this one as sorted out
However, feel free to re-open if you see that is required,