add rate limiting for offline batch jobs, set default bulk size to 500

[Zhangxunmt]

Description

Add rate limiting to offline batch inference and ingestion. Set the default bulk size for batch ingestion to 500(optimal value from benchmark). If the batch job tasks exceeds the limits defined in the settings, ml_limit_exceeded exception will be thrown.

POST /_plugins/_ml/_batch_ingestion{...}
<if exceeding the rate limit>
{
  "error": {
    "root_cause": [
      {
        "type": "m_l_limit_exceeded_exception",
        "reason": "Exceeded maximum limit for BATCH_INGEST tasks. To increase the limit, update the plugins.ml_commons.max_batch_ingestion_tasks setting."
      }
    ],
    "type": "m_l_limit_exceeded_exception",
    "reason": "Exceeded maximum limit for BATCH_INGEST tasks. To increase the limit, update the plugins.ml_commons.max_batch_ingestion_tasks setting."
  },
  "status": 429
}

Update settings for rate limits

PUT _cluster/settings
{
  "persistent": {
    "plugins.ml_commons.offline_batch_inference_enabled": true,
    "plugins.ml_commons.offline_batch_ingestion_enabled": true,
    "plugins.ml_commons.max_batch_inference_tasks": 2,
    "plugins.ml_commons.max_batch_ingestion_tasks": 1,
    "plugins.ml_commons.batch_ingestion_bulk_size": 500
  }
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[rbhavna]

We should add these new settings to netty/jetty layers on AOS 2.17

[Zhangxunmt]

good callout. They will be handled separately when this is backported into the AOS branch.

[jngz-es]

Do we allow users to set a negative number?

[Zhangxunmt]

No the minimum values are 0 actually as defined in the setting.