[1915] Use delay function in waiter

[dazza-codes]

Fix #1915

• use a randomized exponential backoff delay
  for waiter retries by default
• allow a custom callable for waiter retries
• the motivation for this is to avoid API throttle limits for highly concurrent systems

[codecov-io]

Codecov Report

Merging #1921 into develop will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##           develop    #1921   +/-   ##
========================================
  Coverage    92.97%   92.97%           
========================================
  Files           60       60           
  Lines        10797    10807   +10     
========================================
+ Hits         10038    10048   +10     
  Misses         759      759           

Impacted Files	Coverage Δ
botocore/waiter.py	98.76% <100.00%> (+0.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce4ad89...3e8157c. Read the comment docs.

[dazza-codes]

Taggging @kyleknap for review, as this is related to something @kyleknap already reviewed in #1882 - hope you and/or colleagues have a chance to check this out.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[dazza-codes]

Why did I ever bother to submit a PR on this project, just to have a github-bot close it?

[kdaily]

Hi @dazza-codes,

I apologize for the delay in reviewing and responding to this PR. Unfortunately we will not be making these changes to the waiter in this package. Changes to how the waiter works would need to be implemented across all AWS SDKs that support them, as they all share a common implementation. This will require coordination with the rest of the AWS SDK teams as well as service teams.

The current waiter is not designed with these cases in mind. There's no pool for concurrency or mitigation strategy for these kinds of issues, and implementing this appropriately would require more abstraction.

It is an interesting feature request, but not one that has garnered much other attention. I would suggest that we leave the feature request #1915 open, but reframe it around the limitation of the waiter in the case of throttling and concurrency instead of this specific implementation.