Make cluster manager throttling retry delay as dynamic setting

[dhwanilpatel]

Description

• Made retry delay for cluster manager throttling as dynamic settings. We can tune the retries from data node using this settings.
• Made 5s base delay and 30s max delay as default delay settings.

Issues Resolved

#479

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13497/
• CommitID: e10f804
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/13498/
• CommitID: 0bc5018
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/13501/
• CommitID: 914c3f1

[codecov-commenter]

Codecov Report

Merging #6998 (1ad4bb0) into main (ec92b4b) will increase coverage by 0.09%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6998      +/-   ##
============================================
+ Coverage     70.66%   70.75%   +0.09%     
- Complexity    59428    59474      +46     
============================================
  Files          4852     4852              
  Lines        285188   285204      +16     
  Branches      41119    41119              
============================================
+ Hits         201532   201803     +271     
+ Misses        67107    66784     -323     
- Partials      16549    16617      +68     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	93.18% <ø> (ø)
...java/org/opensearch/action/bulk/BackoffPolicy.java	85.39% <100.00%> (ø)
...stermanager/TransportClusterManagerNodeAction.java	78.70% <100.00%> (-6.21%)	⬇️
...h/cluster/service/ClusterManagerTaskThrottler.java	91.08% <100.00%> (+1.43%)	⬆️

... and 495 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[shwetathareja]

@dhwanilpatel the changes look good. Please open an issue to update documentation that base and max delay is configurable now.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/13982/
• CommitID: 1ad4bb0