After upgrade to 2.12.x, sync of "out-of-sync" resources no longer working.

[tromsosec]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Our setup uses Argo-CD applications deployed using ApplicationSets on an app of apps pattern. Applications are multisource, i.e. using git generator and Helm Chart (OCI) + Values (Git repo). We currently do not have Auto-Sync enabled.

On our development environment, our Helm Charts artifacts are created and pushed to the OCI registry using rolling tags with "-dev" suffix, e.g. 0.0.1-dev. We overwrite the artifact with same tag upon new CI build completion.

We are currently running ArgoCD 2.11.4 and after upgrading to version 2.12.x (tried 2.12.1-3), an application running a chart with version 0.0.1-dev and upon the creation of a new chart with same tag version including changes, the changes are detected by Argo (application marked Out-Of-Sync) but the resources that are marked as Out-of-Sync cannot be synced any longer using the "Sync" button. However, if we enable "Auto-Sync" for the application the resources are synced as expected.

Additionally, performing a "Sync" with resource replacement recreates the resources but of previous state, still out of sync.

When the Helm Chart is updated to a different version tag, e.g. 0.0.2-dev there is no issue.
It may be related on how the artifacts are cached. If we restart the Redis pods the first sync works but from there on we have the same above described behaviour.

To Reproduce

1. Create and deploy an ArgoCD application which uses an OCI helm chart and a values file on a git repo.
2. Update the Helm Chart with some changes and push the chart to the OCI registry using the same tag.
3. Confirm that ArgoCD detects new chart (force with Hard Refresh) and application is marked Out-Of-Sync.
4. Try to perform a manual full sync of the application.

Expected behavior

Triggering a Sync using the UI should sync the resources to the latest state and not remained unapplied.

Screenshots

Version

argocd@argo-cd-argocd-application-controller-0:~$ argocd version
argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T11:57:48Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64

Logs

ApplicationController logs when manual resources sync is triggered:

2024-09-17 13:07:15.092	time="2024-09-17T11:07:15Z" level=info msg="updated 'argocd-system/sampleapp' operation (phase: Running)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:15.092	time="2024-09-17T11:07:15Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:,Prune:false,DryRun:false,SyncStrategy:&SyncStrategy{Apply:nil,Hook:&SyncStrategyHook{SyncStrategyApply:SyncStrategyApply{Force:false,},},},Resources:[]SyncOperationResource{SyncOperationResource{Group:,Kind:ConfigMap,Name:sampleapp-sampleappcatalogapi,Namespace:devsecopsref,},SyncOperationResource{Group:apps,Kind:Deployment,Name:sampleapp-sampleappcatalogapi,Namespace:devsecopsref,},},Source:nil,Manifests:[],SyncOptions:[],Sources:[]ApplicationSource{ApplicationSource{RepoURL:registry.com/appsarchitecture/devsecopsref/examples/sampleapp/helmcharts,Path:,TargetRevision:0.2.0,Helm:&ApplicationSourceHelm{ValueFiles:[$values/lab/values.yaml],Parameters:[]HelmParameter{},ReleaseName:sampleapp,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:false,IgnoreMissingValueFiles:false,SkipCrds:false,ValuesObject:nil,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:sampleapp-umbrella-chart/sampleapp,Ref:,},ApplicationSource{RepoURL:https://gitrepo/appsarchitecture/devsecopsref/examples/sampleapp/deployment/sampleapp-lab.git,Path:,TargetRevision:main,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:values,},},Revisions:[0.2.0 194a0d0df583a4f098519ecdea9e1d212e34ddf3],} {redacted-user false} [] {0 nil}}" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:15.092	time="2024-09-17T11:07:15Z" level=info msg="Comparing app state (cluster: https://cluster/redacted, namespace: devsecopsref)" application=argocd-system/sampleapp
2024-09-17 13:07:15.121	time="2024-09-17T11:07:15Z" level=info msg="GetRepoObjs stats" application=argocd-system/sampleapp build_options_ms=0 helm_ms=25 plugins_ms=0 repo_ms=0 time_ms=27 unmarshal_ms=1 version_ms=0
2024-09-17 13:07:15.192	time="2024-09-17T11:07:15Z" level=info msg=Syncing application=argocd-system/sampleapp skipHooks=true started=false syncId=00028-jmCBS
2024-09-17 13:07:15.276	time="2024-09-17T11:07:15Z" level=info msg="Tasks (dry-run)" application=argocd-system/sampleapp syncId=00028-jmCBS tasks="[Sync/0 resource /ConfigMap:devsecopsref/sampleapp-sampleappcatalogapi obj->obj (,,), Sync/0 resource apps/Deployment:devsecopsref/sampleapp-sampleappcatalogapi obj->obj (,,)]"
2024-09-17 13:07:15.277	time="2024-09-17T11:07:15Z" level=info msg="Applying resource ConfigMap/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:15.630	time="2024-09-17T11:07:15Z" level=info msg="Applying resource Deployment/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:15.745	time="2024-09-17T11:07:15Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd-system/sampleapp syncId=00028-jmCBS
2024-09-17 13:07:15.745	time="2024-09-17T11:07:15Z" level=info msg="Applying resource ConfigMap/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=none manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:16.207	time="2024-09-17T11:07:16Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'configmap/sampleapp-sampleappcatalogapi unchanged'" application=argocd-system/sampleapp kind=ConfigMap name=sampleapp-sampleappcatalogapi namespace=devsecopsref phase=Sync syncId=00028-jmCBS
2024-09-17 13:07:16.207	time="2024-09-17T11:07:16Z" level=info msg="Applying resource Deployment/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=none manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:16.264	time="2024-09-17T11:07:16Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'deployment.apps/sampleapp-sampleappcatalogapi unchanged'" application=argocd-system/sampleapp kind=Deployment name=sampleapp-sampleappcatalogapi namespace=devsecopsref phase=Sync syncId=00028-jmCBS
2024-09-17 13:07:16.264	time="2024-09-17T11:07:16Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd-system/sampleapp syncId=00028-jmCBS
2024-09-17 13:07:16.264	time="2024-09-17T11:07:16Z" level=info msg="sync/terminate complete" application=argocd-system/sampleapp duration=1.071493944s syncId=00028-jmCBS
2024-09-17 13:07:16.295	time="2024-09-17T11:07:16Z" level=info msg="updated 'argocd-system/sampleapp' operation (phase: Succeeded)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.295	time="2024-09-17T11:07:16Z" level=info msg="Partial sync operation to  succeeded" application=sampleapp dest-namespace=devsecopsref dest-server="https://cluster/redacted" reason=OperationCompleted type=Normal
2024-09-17 13:07:16.308	time="2024-09-17T11:07:16Z" level=info msg="Refreshing app status (controller refresh requested), level (3)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.309	time="2024-09-17T11:07:16Z" level=info msg="Comparing app state (cluster: https://cluster/redacted, namespace: devsecopsref)" application=argocd-system/sampleapp
2024-09-17 13:07:16.330	time="2024-09-17T11:07:16Z" level=info msg="No status changes. Skipping patch" app-namespace=argocd-system app-qualified-name=argocd-system/argocd-appsetsroot application=argocd-appsetsroot project=argocd-appsroot
2024-09-17 13:07:16.765	time="2024-09-17T11:07:16Z" level=info msg="GetRepoObjs stats" application=argocd-system/sampleapp build_options_ms=0 helm_ms=230 plugins_ms=0 repo_ms=0 time_ms=456 unmarshal_ms=224 version_ms=0
2024-09-17 13:07:16.856	time="2024-09-17T11:07:16Z" level=info msg="Skipping auto-sync: most recent sync already to " app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.884	time="2024-09-17T11:07:16Z" level=info msg="Update successful" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.885	time="2024-09-17T11:07:16Z" level=info msg="Reconciliation completed" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp comparison-level=3 dedup_ms=0 dest-name=lab dest-namespace=devsecopsref dest-server="https://cluster/redacted" diff_ms=68 git_ms=456 health_ms=2 live_ms=0 patch_ms=27 project=devsecopsref setop_ms=0 settings_ms=0 sync_ms=0 time_ms=576

Kubernetes ArgoCD Deployment

kubectl get pods
NAME                                                       READY   STATUS    RESTARTS   AGE
argo-cd-argocd-application-controller-0                    1/1     Running   0          130m
argo-cd-argocd-applicationset-controller-7d6bc9759-vd9ms   1/1     Running   0          130m
argo-cd-argocd-dex-server-d774895d5-zl82b                  1/1     Running   0          145m
argo-cd-argocd-notifications-controller-6dbd674d94-jfmg4   1/1     Running   0          145m
argo-cd-argocd-repo-server-78d45c48d-vpmpf                 1/1     Running   0          129m
argo-cd-argocd-repo-server-78d45c48d-w7qjt                 1/1     Running   0          129m
argo-cd-argocd-server-b4496dcbb-2fjfn                      1/1     Running   0          129m
argo-cd-argocd-server-b4496dcbb-4s6mj                      1/1     Running   0          129m
argo-cd-redis-ha-haproxy-6ffdb86584-j9466                  1/1     Running   0          118m
argo-cd-redis-ha-haproxy-6ffdb86584-jqlv8                  1/1     Running   0          118m
argo-cd-redis-ha-haproxy-6ffdb86584-jzql5                  1/1     Running   0          118m
argo-cd-redis-ha-server-0                                  3/3     Running   0          117m
argo-cd-redis-ha-server-1                                  3/3     Running   0          115m
argo-cd-redis-ha-server-2                                  3/3     Running   0          114m