fix: release reconcile modified resource error

[0x0013]

Start With Why

For our Release reconciliation loop, we are observing intermittent errors such as:

theatre-release-manager-5b49cc44d7-mwc8d manager ts=2026-02-06T15:53:12.120989983Z caller=logr@v1.4.3/logr.go:280 msg="Operation cannot be fulfilled on releases.deploy.crd.gocardless.com \"elozev-playground-l2l7m\": the object has been modified; please apply your changes to the latest version and try again" component=Release request="{elozev-playground-l2l7m elozev}" eventType=DontRecord event=ReconcileError error="Operation cannot be fulfilled on releases.deploy.crd.gocardless.com \"elozev-playground-l2l7m\": the object has been modified; please apply your changes to the latest version and try again"

Though these errors self-resolve within the next 1-2 loops, it's obviously not ideal to have them.

Rationale

With some assistance from LLM, we have a proposed cause and a potential fix.

Cause

The conflict error ("the object has been modified") is a Kubernetes optimistic concurrency error. It occurs because CreateOrUpdate in pkg/recutil/reconcile.go:150-188 uses the cached client for the re-GET but sends the Status Update directly to the API server.

The cache (backed by controller-runtime informers) lags behind the API server. The re-GET at step 5 was designed to get the latest version, but it reads from the stale cache, not the API server directly.

Potential fix

Use MergePatch for status updates (most robust): Replace c.Status().Update() with c.Status().Patch() using a merge patch. Patches are more tolerant of concurrent modifications since they don't require an exact resourceVersion match.

Considerations

This PR is primarily meant to test whether the solution works. I'm not entirely convinced about the fix as-is, because it modifies the CreateOrUpdate logic and takes a different approach for Update and StatusUpdate.

Furthermore, the approach makes the behavior of the generic DiffFunc partially redundant - the function modifies the existing copy of the state, with the intention that it will be sent to update, however in the case of StatusUpdate result, we ignore the updated existing object.

If the fix works, we should consider options to better integrate it into the generic functionality of CreateOrUpdate

Results

After preliminary test, I don't see any of the previous "the object has been modified" errors.