[Question / Bug] kustomize-controller applying change to field managed by another manager

[choppedpork]

Hello,

I'm currently working with a colleague on migrating a few outstanding workloads from Flux 1 to Flux 2 and while it's so far going swimmingly we've hit quite an interesting issue causing constant drift and a back-and-forth change every Kustomization interval.

The issue is with a validating webhook (FWIW originally created by Flux 1) which looks like this (excluding a whole bunch of fields for brevity but including managed fields):

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
(...)
  managedFields:
  - apiVersion: admissionregistration.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app: {}
          f:istio: {}
          f:istio.io/rev: {}
          f:kustomize.toolkit.fluxcd.io/name: {}
          f:kustomize.toolkit.fluxcd.io/namespace: {}
          f:release: {}
      f:webhooks:
        k:{"name":"rev.validation.istio.io"}:
          .: {}
          f:admissionReviewVersions: {}
          f:clientConfig:
            f:service:
              f:name: {}
              f:namespace: {}
              f:path: {}
          f:name: {}
          f:objectSelector: {}
          f:rules: {}
          f:sideEffects: {}
    manager: kustomize-controller
    operation: Apply
    time: "2023-05-23T15:10:14Z"
  - apiVersion: admissionregistration.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:webhooks:
        k:{"name":"rev.validation.istio.io"}:
          f:clientConfig:
            f:caBundle: {}
          f:failurePolicy: {}
    manager: pilot-discovery
    operation: Update
    time: "2023-05-23T15:10:14Z"
(...)
webhooks:
- (...)
  failurePolicy: Fail

The webhook itself comes from Istio and includes a comment on this very field: https://github.com/istio/istio/blob/master/manifests/charts/istio-control/istio-discovery/templates/validatingwebhookconfiguration.yaml#L41-L44

    # Fail open until the validation webhook is ready. The webhook controller
    # will update this to `Fail` and patch in the `caBundle` when the webhook
    # endpoint is ready.
    failurePolicy: Ignore

This all makes perfect sense - when initially deployed by Flux the policy is set to Ignore, once the controller is ready it updates the webhook to Fail and remains like that ever since. Except... it doesn't, because kustomize-controller is attempting to change it at every interval (and then it's subsequently changed back by the controller - you can see in the above snippet that the Update time in both entries in managedFields is identical). Having encountered fluxcd/flux2#2386 previously during another migration I thought that maybe this is something that can be solved via a manual patch of managed fields but subsequently realised that the failurePolicy field is already owned by the pilot-discovery entity.

I've only stumbled upon managed fields as part of the aforementioned issue so it's not something I'm particularly knowledgeable in ... Is my understanding of how this is meant to be correct at all? Shouldn't kustomize-controller be ignoring any changes made to failurePolicy because that field is managed by another entity?

Thanks!

[choppedpork]

This seems similar to the issue talked about here except in the case of the failurePolicy we do want to set it to a particular value initially.

[stefanprodan]

This is how Flux was designed to work, if a field is set in Git, the value will always be overwritten. Docs here: https://fluxcd.io/flux/components/kustomize/kustomization/#customizing-reconciliation