Multiple Resolver resolution results in infinite patching

[sdischer-sap]

What happened?

We found a reproducable inconsistancy when persisting resolved references. Basically whenever you have more then one reference and one of them is resolved in a later reconcilation loop then the first one patching this newly resolved value will purge previously resolved values at the same time, which will result in an endless loop of patching different values back and forth.

Debugging this issue showed that the issue must lie in the way that the change is send as a server-sided apply in this line:
https://github.com/crossplane/crossplane-runtime/blob/master/pkg/reconciler/managed/api.go#L193

Just FYI, doing the same patch from kubectl using a merge patch instead worked as expected.

How can we reproduce it?

A reproducable case can be easily reproduce in this example provider: https://github.com/mirzakopic/provider-xp-bug
You will find a mytype Kind having two references towards other kinds called sometype and othertype.
The issue appears when you first apply the mytype with only one reference and reapply another version with both references being set.

You can reproduce that with the provider using this flow:

1. (Setup the provider in a local environment)
2. kubectl apply -f examples/sample/mytype.yaml
3. kubectl apply -f examples/sample/mytype_added_ref.yaml

Screenflow of endless resolving:

What environment did it happen in?

Crossplane version:
1.16 and 1.15 (but should apply to older versions as well(

[github-actions]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[MrVinkel]

We are seeing the same issue.

We have copied the resolver from crossplane-runtime and used apimachinery ExtractInto to extract the managed fields and added them to the JSON patch. This makes it so the patch always contains all the fields the reference resolver owns plus the resolved refs and stops the infinite patching.

I'm not sure this is the correct way or if there is a better way to do it.

[TehilaTheStudent]

hey, maybe you can paste your solution here? i didnt suceed reproducing the solutions you described here, that will help a lot.., thanks!

[TehilaTheStudent]

if im not wrong, the issue happens also when there is only one ref, but you specify matchLabels,
than, in patching it patches 2 fields: filedIdRef and fieldId, if you come to scenario that only one of them needs to be changed - lets say the name if the managedResource changed but its id not, than again the infinite patching loop begins..

[TehilaTheStudent]

As documented in the Kubernetes docs on https://kubernetes.io/docs/reference/using-api/server-side-apply/#using-server-side-apply-in-a-controller

When using Server-Side Apply, always include all fields in the object that the controller cares about.

Fields not included in the apply request will be pruned if the controller previously owned them. This is by design — see the implementation https://github.com/kubernetes-sigs/structured-merge-diff/blob/master/merge/update.go#L256

In this issue's case, when the resolver patches only the newly resolved reference, SSA prunes the previously resolved reference (since it's no longer in the patch). The next reconcile re-resolves and patches it back, pruning the other — causing the infinite loop.