Object has been modified while testing out a ClusterExtension

[itroyano]

E1021 20:42:57.252397 1 controller.go:316] "Reconciler error" err="error updating status: Operation cannot be fulfilled on clusterextensions.olm.operatorframework.io \"metrics-server\": the object has been modified; please apply your changes to the latest version and try again" controller="clusterextension" controllerGroup="olm.operatorframework.io" controllerKind="ClusterExtension" ClusterExtension="metrics-server" namespace="" name="metrics-server" reconcileID="ef934a9a-f494-462e-9d24-1235a1435f9b"

Code ref: https://github.com/operator-framework/operator-controller/blob/main/internal/controllers/clusterextension_controller.go#L132

Suggested approaches:

• Shorter term: return a re-queue or re-fetch the CE object here to make sure the resourceVersion is in sync [1]
• Longer term: look into Server Side Apply [2] [3] to knock out such issues.

[1] https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes
[2] https://kubernetes.io/docs/reference/using-api/server-side-apply/
[3] kubernetes-sigs/controller-runtime#347

[itroyano]

/assign

[joelanford]

Can you include the version of operator-controller and more logging context.

I'm not sure that SSA is a sufficient solution for this problem. The error is telling us that the status that we're trying to write doesn't actually "go with" the current state of the object, which has changed since we last read it. In order to write the correct status, we need to requeue and fetch again anyway.

Having said that, I would only expect this error to happen in the following scenario

1. We start reconciling a ClusterExtension
2. Some other user or component updates the ClusterExtention
3. We finish reconciling and attempt to update status (or add/remove finalizers)

In that scenario, I would expect us to requeue and try again (which is what we do now). But if we're seeing this error in other scenarios, that points to an issue in the way that we are sending updates to the apiserver for an object.