fix(workflow/sync): use RWMutex to prevent concurrent map access

[ryancurrah]

This resolves a fatal "concurrent map iteration and map write" error by synchronizing reads and writes to syncLockMap with a read-write mutex. This change has been running in our environment for 7 days (8,764 workflows) without reoccurrence.

Fixes #14300

Motivation

To stop fatal panics when reading and writing to the syncLockMap map at the same time.

Modifications

• Added a read lock in the CheckWorkflowExistence function.
• Replaced the exclusive lock with a read-write mutex.
• For functions that only read syncLockMap, a read lock is used to allow concurrent reads safely.

Verification

• Built the Argo Workflows controller image with make workflow-controller-image.
• Pushed it to our Artifactory container registry.
• Deployed in a non-production cluster (running production-like workloads in “shadow mode”).
• Monitored logs for 7 days (8,764 workflows) with alerts on the known error message.
• Observed no recurrence of the concurrency issue.

Documentation

N/A

[Copilot]

Pull Request Overview

This PR addresses a fatal "concurrent map iteration and map write" error by switching from a basic mutex to a read-write mutex in order to better synchronize access when workflows are checked, released, or released all at once.

• The Manager struct now uses a sync.RWMutex.
• CheckWorkflowExistence now acquires a read lock.
• The Release and ReleaseAll methods have been updated to use read locks.

Comments suppressed due to low confidence (1)

workflow/sync/sync_manager.go:342

• If the ReleaseAll method modifies the syncLockMap or its contents, consider using a write lock to ensure safe concurrent modifications.

sm.lock.RLock()

[Joibel]

/cherry-pick release-3.6