Model & user level throttling

[b4sjoo]

Description

This PR should enable model and user level rate limiting on TextEmbedding model and all Remote model.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

Is this PR ready to review? Spotless missing. Is that 111 files? Or other commits also added here?

[b4sjoo]

Is this PR ready to review? Spotless missing. Is that 111 files? Or other commits also added here?

Main codes are ready to review, just some UTs are missing. This PR including some UT names refactor, but overall they are only about the throttling feature itself.

[b4sjoo]

Yeah it's a super large PR...

[Zhangxunmt]

added a few comments from user experience and data model in the OS index. Do you have a quip document to track all the APIs usages and examples that will be added in the PR?

[b4sjoo]

added a few comments from user experience and data model in the OS index. Do you have a quip document to track all the APIs usages and examples that will be added in the PR?

Addressed. For the API usage will added to the same quip we used to demo later today.

[Zhangxunmt]

This can be merged with 2 major concerns from my view:

1. There will be data redundancy in the Ml-Model because both user/model throttling are bind within the Ml-Model. It's not obvious that we can decouple them easily in the future.
2. The APIs added in the PR should be all optional for users to run ml-commons. There shouldn't be any required setup for throttling unless someone needs to use throttling.

[b4sjoo]

This can be merged with 2 major concerns from my view:

There will be data redundancy in the Ml-Model because both user/model throttling are bind within the Ml-Model. It's not obvious that we can decouple them easily in the future.
The APIs added in the PR should be all optional for users to run ml-commons. There shouldn't be any required setup for throttling unless someone needs to use throttling.

We can further discuss on the first one. For the second one, at this time all throttling related apis are optional for our user. The default behavior is pass for all requests (same as the previous).

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1800-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 50788de0bcf7a9509da6d2dabb23aab59d5a843d
# Push it to GitHub
git push --set-upstream origin backport/backport-1800-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1800-to-2.x.