Add configurable QPS and burst settings for kube API client

[ronk21runai]

What this PR does / why we need it:
Currently, the default configuration of QPS (20) and Burst (30) is configured by the controller runtime defaults, which are not adjustable by the user. This PR allows users to fine-tune these values, improving the controller's performance.
Introduce new flags to configure QPS and Burst for the Kubernetes API client, enabling better control over API rate limits.

*Proposed Changes
This PR introduces two new argument flags:

flag.IntVar(&clientQps, "kube-api-qps", 20, "QPS indicates the maximum QPS to the master from this client.")
flag.IntVar(&clientBurst, "kube-api-burst", 30, "Maximum burst for throttle.")

These flags allow users to configure API rate limits dynamically instead of relying on the default values.

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 13135248279

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 13092942826:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[tenzen-y]

@ronk21runai Could you rebase this PR top on the release-1.9 branch? We has already been removed v1 codes from the master branch.

[tenzen-y]

Mostly lgtm
Thank you

@ronk21runai Once you rebase and address my comment, we can contain this in the release-1.9.

[tenzen-y]

Suggested change

	cfg.QPS = float32(clientQps)
	cfg.Burst = clientBurst
	cfg.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(float32(clientQps), clientBurst)

Due to controller-runtime specification, IIUC, we need to specify those parameters throughout the RateLimiter.

[andreyvelich]

Thank you for this great contribution!
I am wondering, should we support it in TrainJob controller as well ?

[tenzen-y]

Thank you for this great contribution! I am wondering, should we support it in TrainJob controller as well ?

Ideally, we want to support those and manager specific parameters in the Config API for v2

[andreyvelich]

I agree, I will create an issue to add Config API support into Kubeflow Trainer V2.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tenzen-y]

otherwise lgtm

[tenzen-y]

could you move this to second block?

[andreyvelich]

Thanks @ronk21runai!
Please can you create tracking issue to introduce QPS setting for Kubeflow Trainer V2 ?
/lgtm
/assign @tenzen-y