fix: Mitigate the bug where items are re-added constantly to the workqueue. #1193

[MarkSRobinson]

There is a deep bug where items are added to the rollouts workqueue constantly. This is a problem because there is an exponential back-off for items so each add extends the back-off by a factor of two. The backoff maxes out at 16.6 minutes.

This fix will prevent Argo Rollouts from hanging for up to 16 minutes at a time if this case happens. This change reduces the maximum back-off to 10 seconds.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

[codecov]

Codecov Report

Merging #1243 (559d6fb) into master (d9d1237) will increase coverage by 0.02%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master    #1243      +/-   ##
==========================================
+ Coverage   81.40%   81.42%   +0.02%     
==========================================
  Files         106      106              
  Lines        9527     9531       +4     
==========================================
+ Hits         7755     7761       +6     
+ Misses       1251     1250       -1     
+ Partials      521      520       -1     

Impacted Files	Coverage Δ
rollout/controller.go	78.35% <66.66%> (-0.10%)	⬇️
...ctl-argo-rollouts/viewcontroller/viewcontroller.go	71.84% <100.00%> (ø)
rollout/trafficrouting/istio/controller.go	45.73% <100.00%> (+2.32%)	⬆️
utils/controller/controller.go	82.92% <100.00%> (+0.13%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9d1237...559d6fb. Read the comment docs.

[jessesuen]

This is a great find! Could you describe the scenario where this happens? I'm just surprised we have not come across this.

Also, can you fix linting errors? After that it looks good to merge.

[MarkSRobinson]

I'm not entirely sure since it can be hard to reproduce, but the big thing is high pod count (>20) and analysis runs that are frequent and don't terminate. So every 10s for analysis checks. It also correlates with long deployment times (>10m)

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information