ArgoCD application mananaged resources fail to delete with finalizer set to resources-finalizer.argocd.argoproj.io

[sholad]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I created a kyverno policy to update my argocd application with the finalizer "resources-finalizer.argocd.argoproj.io". The application was created and the finalizer was present in the application's live manifest. However, when I deleted the application, the child resources managed was deleted but the app was stuck in Deletion.

To Reproduce
=====Kyverno policy====

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: argocd-policies
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    policies.kyverno.io/title: ArgoCD Policies
    policies.kyverno.io/category: Other
    policies.kyverno.io/subject: ArgoCD
spec:
  background: true
  rules:
    - name: add-resources-finalizer
      match:
        any:
          - resources:
              kinds:
                - argoproj.io/*/Application
              names:
                - test-app
             
      context:
        - name: finalizers_array
          variable:
            jmesPath: request.object.metadata.finalizers
            default: []
      preconditions:
        all:
          - key: "{{ length(finalizers_array) }}"
            operator: Equals
            value: 0
      mutate:
        patchStrategicMerge:
          metadata:
            finalizers:
              - resources-finalizer.argocd.argoproj.io

====Application ====

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-app
  namespace: argocd
spec:
  destination:
    server: {{ .Values.appDestinationServer }}
    namespace: test-policy
  source:
    path: kustomize/overlays/{{ .Values.clusterName }}/operations/policy-test
    repoURL: {{ .Values.repoURL }}
    targetRevision: {{ .Values.targetRevision }}
  project: operations
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Push the manifests to the gitops repo.
Wait for argocd to create the application and resources.
Comment out or remove the application from the gitops repo. To delete the app.

Expected behavior
Application stuck in Deleting state

Expected the application to be deleted, including the managed resources.

Version
v2.12.4+27d1e64

[crenshaw-dev]

Can you create a minimal reproduction of the issue? It's difficult to know what went wrong without access to the manifests of the resources managed by the app an to the k8s API for the cluster where they were deployed.

[sholad]

@crenshaw-dev The issue can be simulated with any resource(Deployment, CM etc).
The resources were deleted but the app wasn't.

[crenshaw-dev]

Can you post the Application's status field?

[sholad]

[sholad]

status:
  controllerNamespace: argocd
  health:
    status: Missing
  history:
    - deployStartedAt: '2024-10-08T18:52:40Z'
      deployedAt: '2024-10-08T18:52:40Z'
      id: 0
      initiatedBy:
        automated: true
      revision: XXXXX
      source:
        path: XXXXX
        repoURL: XXXXX
        targetRevision: XXXXX
  operationState:
    finishedAt: '2024-10-08T18:52:41Z'
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        limit: 5
      sync:
        prune: true
        revision: XXXXX
        syncOptions:
          - CreateNamespace=true
    phase: Succeeded
    startedAt: '2024-10-08T18:52:40Z'
    syncResult:
      resources:
        - group: apps
          hookPhase: Running
          kind: Deployment
          message: deployment.apps/test-policy created
          name: test-policy
          namespace: test-policy
          status: Synced
          syncPhase: Sync
          version: v1
      revision: XXXXX
      source:
        path: XXXXX
        repoURL: XXXXX
        targetRevision: main
  reconciledAt: '2024-10-09T16:09:59Z'
  resources:
    - group: apps
      health:
        status: Missing
      kind: Deployment
      name: test-policy
      namespace: test-policy
      status: OutOfSync
      version: v1
  sourceType: Kustomize
  summary: {}
  sync:
    comparedTo:
      destination:
        namespace: test-policy
        server: XXXXX
      source:
        path: XXXXX
        repoURL: XXXXX
        targetRevision: main
    revision: XXXXX
    status: OutOfSync

[crenshaw-dev]

I just tried to reproduce this with the guestbook app. I used kubectl edit to add the finalizer and kubectl delete to delete the app. The Deployment was deleted, and the App disappeared.

I'd recommend checking your controller logs.

[sholad]

@crenshaw-dev The finalizer was added using a Kyverno Policy.

[agaudreault]

@sholad can you validate that your kyverno policy only ran during the resource creation. If it runs at every update, whenever argo tries to remove the finalizer because it deletes the resources, kyverno may add it back.