Duplicated notifications based on argocd-notifications-controller replica count

[francesco-furnari]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

The Argo CD notifications are being duplicated based on the number of replicas for the argocd-notifications-controller. Specifically, when the argocd-notifications-controller is scaled to 1 replica, we receive the expected single notification for events. However, scaling to 2 replicas results in each notification being sent twice, and this duplication increases proportionally with the number of replicas.

To Reproduce

1. Install Argo CD and the notifications component using the HA manifest from the notifications catalog.
2. Configure notifications (e.g., for Google Chat or Webhook).
3. Scale the argocd-notifications-controller deployment to 1 replica and observe the number of notifications sent for an event.
4. Scale the argocd-notifications-controller deployment to 2 replicas and observe the number of notifications sent for the same type of event.
5. Repeat step 4 with more replicas and note the number of duplicate notifications.

Expected behavior

Irrespective of the number of argocd-notifications-controller replicas, only one notification should be sent per event.

Version

argocd: v2.10.1+a79e0ea
BuildDate: 2024-02-14T17:56:39Z
GitCommit: a79e0eaca415461dc36615470cecc25d6d38cefb
GitTreeState: clean
GoVersion: go1.21.7
Compiler: gc
Platform: linux/amd64
argocd-server: v2.10.0+2175939

Logs
Here you can find the logs of the two notifications. I have filtered out some unnecessary values. If you compare the logs, the only difference is a different cardId.

From the first notification controller

time="2024-02-28T10:26:32Z" level=info msg="Start processing" resource=argocd/core
time="2024-02-28T10:26:32Z" level=debug msg="SyncStatus up-to-date (FinishedAt=2024-02-28 10:26:32 +0000 UTC, ReconciledAt=2024-02-28 10:26:32 +0000 UTC, Observed=0001-01-01 00:00:00 +0000 UTC" app=core
time="2024-02-28T10:26:32Z" level=info msg="Trigger on-deployed result: [{[0].y7b5sbwa2Q329JYH[...]-fBs 4fb8936e478e[...]26712d78c348575e [app-deployed] true}]" resource=argocd/core
time="2024-02-28T10:26:32Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH[...]-fBs' to '{googlechat }' using the configuration innamespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=error msg="Failed to notify recipient {googlechat } defined in resource argocd/core: error creating client to webhook: no Google chat webhook configured for recipient  using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH[...]-fBs' to '{googlechat argoWebhookTest}' using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=debug msg="Sending request: POST [.filtered.]  \"cardId\":\"0f6394ae-6441-4f19-a54c-e3a4335e613e\"}]}" service=googlechat
time="2024-02-28T10:26:34Z" level=debug msg="Received response: HTTP/2.0 200 OK [.filtered.]"
time="2024-02-28T10:26:34Z" level=debug msg="Notification argoWebhookTest was sent using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:34Z" level=info msg="Processing completed" resource=argocd/core

From the second notification controller

time="2024-02-28T10:26:32Z" level=info msg="Start processing" resource=argocd/core
time="2024-02-28T10:26:32Z" level=debug msg="SyncStatus up-to-date (FinishedAt=2024-02-28 10:26:32 +0000 UTC, ReconciledAt=2024-02-28 10:26:32 +0000 UTC, Observed=0001-01-01 00:00:00 +0000 UTC" app=core
time="2024-02-28T10:26:32Z" level=info msg="Trigger on-deployed result: [{[0].y7b5sbwa2Q329JYH[...]-fBs 4fb8936e478e[...]26712d78c348575e [app-deployed] true}]" resource=argocd/core
time="2024-02-28T10:26:32Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH[...]-fBs' to '{googlechat }' using the configuration innamespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=error msg="Failed to notify recipient {googlechat } defined in resource argocd/core: error creating client to webhook: no Google chat webhook configured for recipient  using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=info msg="Sending notification about condition 'on-deployed.[0].y7b5sbwa2Q329JYH[...]-fBs' to '{googlechat argoWebhookTest}' using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:33Z" level=debug msg="Sending request: POST [.filtered.],\"cardId\":\"e6ca11a7-d335-4c23-9b57-cb08b36e9c1f\"}]}" service=googlechat
time="2024-02-28T10:26:34Z" level=debug msg="Received response: HTTP/2.0 200 OK\r\n [.filtered.]"
time="2024-02-28T10:26:34Z" level=debug msg="Notification argoWebhookTest was sent using the configuration in namespace argocd" resource=argocd/core
time="2024-02-28T10:26:34Z" level=info msg="Processing completed" resource=argocd/core