enable customize http client parameters

[zane-neo]

Description

Currently remote inference using CloseableHttpClient with default configurations both for AWS protocol and http protocol, the drawback is customer can't configure the client base on their own use case. Sometimes issue can be found thrown from httpclient, this feature enables customer to configure their own parameters for httpclient.

Issues Resolved

#1470
#1537

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (693a53c) 80.28% compared to head (347fbfc) 80.17%.
Report is 17 commits behind head on 2.x.

❗ Current head 347fbfc differs from pull request most recent head b5f2ed4. Consider uploading reports for the commit b5f2ed4 to get more accurate results

Files	Patch %	Lines
...engine/algorithms/remote/AwsConnectorExecutor.java	0.00%	13 Missing ⚠️
...ine/algorithms/remote/RemoteConnectorExecutor.java	0.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #1558      +/-   ##
============================================
- Coverage     80.28%   80.17%   -0.12%     
- Complexity     2526     2531       +5     
============================================
  Files           199      200       +1     
  Lines         10001    10056      +55     
  Branches       1002     1004       +2     
============================================
+ Hits           8029     8062      +33     
- Misses         1499     1522      +23     
+ Partials        473      472       -1     

Flag	Coverage Δ
ml-commons	80.17% <70.68%> (-0.12%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dhrubo-os]

What if customer wants timeout = 60 ms for remote model A, and timeout = 80 ms for remote model B?

@austintlee do you think this could be a valid use case?

May be customer can provide these parameters during setting up the connector? We'll have some default values, but customer can provide their expected values also if they want?

[zane-neo]

60ms and 80ms may not a perfect example but I think this is a valid use case, different LLMs have different latency performance, I agree that we can add these httpclient configurations to connector. @austintlee please share your thoughts as well.

[Zhangxunmt]

Why not include these three params in the request body of the "Predict" call to each LLM? If they are absent in the request, we can use the default values like these settings.

[zane-neo]

This can be implemented but we'd better make these changes in a different PR, for this PR the target is to enable customer to set these parameters to avoid exceptions.

[ylwu-amzn]

@zane-neo So we have three options now

1. Add a cluster setting like what this PR does
2. Add these parameters to connector configuration, from @dhrubo-os
3. User can set these parameters in request body, from @Zhangxunmt

For my understanding, we already have a lot of cluster settings, maybe let's not add more cluster settings? Seems adding to connector level is more flexible, and if we add to connector parameters, we can also support option3 by default.

[dhrubo-os]

+1 on this.

[zane-neo]

Make sense, this configuration is only working on remote inference case, we can put these in the connector configuration. The only drawback is user has to configure this for each connector which is kind of verbose.

[Zhangxunmt]

Why not include these three params in the request body of the "Predict" call to each LLM? If they are absent in the request, we can use the default values like these settings.