No diff detected in maps with ServerSideDiff enabled

[jrcast]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Drift in maps part of resources monitored by ArgoCD applications is not detected when server-side diff is enabled (controller.diff.server.side: true in argocd-cmd-params-cm), This is happening with multiple resource types. So far, I have reproduced this using (k8s services, workflowtemplates.argoproj.io, CRDs)

To Reproduce

1. Enable server-side diff + restart argocd components.
2. Using ArgoCD UI or kubectl, edit a K8s service that is monitored by an ArgoCD application. i.e. Add an extra port to the list.
3. In the ArgoCD UI, press refresh/hard-refresh and Sync.
4. Verify ArgoCD live manifest shows the change applied, desired manifest doesn't. The diff tab is blank (no diff detected)

Desired manifests:

apiVersion: v1
kind: Service
metadata:
  {...}
spec:
  ports:
    - name: service-port
      port: 80
      protocol: TCP
      targetPort: service-port
  selector:
  {...}
  type: ClusterIP

Live Manifest: (Note the extra port. This was added manually)

apiVersion: v1
kind: Service
metadata:
  {...}
spec:
  ports:
    - name: service-port
      port: 80
      protocol: TCP
      targetPort: service-port
    - name: shouldntbehere    # <<<< THIS 
      port: 8080
      protocol: TCP
      targetPort: service-port
  selector:
  {...}
  type: ClusterIP

The same can be observed in other resource types.

Expected behavior
ArgoCD detects the drift, marks the application "Out Of Sync", and (if Self-Heal is enabled) fixes the drift.

Screenshots
Live Manifests

Desired Manifest

Diff

Version

argocd-server: v2.10.7+b060053
  BuildDate: 2024-04-15T09:05:25Z
  GitCommit: b060053b099b4c81c1e635839a309c9c8c1863e9
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: 5.1.1 unknown
  Helm Version: v3.13.1+gv3.13.1
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs
Nothing out of the ordinary here.

time="2024-04-30T22:23:27Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" appl
ication=argocd/example-app

time="2024-04-30T22:23:27Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespa
ce: myns)" application=argocd/example-app

time="2024-04-30T22:23:27Z" level=info msg="GetRepoObjs stats" application=argocd/example-app build_options_ms=0 
helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=10 unmarshal_ms=10 version_ms=0

time="2024-04-30T22:23:27Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/
example-app

time="2024-04-30T22:23:27Z" level=info msg="Update successful" application=argocd/example-app

time="2024-04-30T22:23:27Z" level=info msg="Reconciliation completed" application=argocd/example-app dedup_ms=0 d
est-name= dest-namespace=myns dest-server="https://kubernetes.default.svc" diff_ms=2 fields.level=2 git_m
s=11 health_ms=1 live_ms=3 patch_ms=15 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=106

[zswanson]

We are seeing the same issue on both 2.10.6 and 2.11.0 except we have server-side-diff disabled (using server-side-apply though). Desired manifest shows volumes/mounts to be removed, live manifest has the volumes/mounts but argo shows no diff and applies no change.

[nielsole]

What @zswanson wrote was unrelated to argocd (kubectl diff also showed an empty diff due to strategic merging) and likely an issue with a third party client previously not handling field manager correctly.
Still a bit weird to me that Argo didn't detect that the application was out of sync even with the field manager was broken.