br,lightning: S3 URL support assuming role

[dsdashun]

What problem does this PR solve?

Issue Number: close #36891

Problem Summary:

What is changed and how it works?

• Added two query parameters for S3 URL in external storage: role-arn and external-id
• When creating an S3 client, if role-arn is set, assume the given role to access the S3 objects.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  • write a main program to construct a Lightning prechecker using a config
  • set a role in AWS IAM with proper trust entity
  • In the config, set the data-source-url to an S3 URL with 'role-arn' and 'external-id' set. For example: s3://my-testing-data/my-dir/?role-arn=arn:aws:iam::88888888:role%2Ftest-role&external-id=abcd1234
  • run the program, the check should not be blocked on 'Access Denied' error
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add query parameters for S3 external storage URL, in order to support accessing the S3 data in another account by assuming a given role

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• D3Hunter
• Ehco1996

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[dsdashun]

/run-integration-br-test

[sre-bot]

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/482091951d582a12c2bd818f487534aae97b3424

[dsdashun]

/run-integration-br-test

[Ehco1996]

LGTM

[dsdashun]

br integration test passed: https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_integration_br_test/detail/tidb_ghpr_integration_br_test/10092/pipeline

[D3Hunter]

rest lgtm

[D3Hunter]

why this pr need to update kvproto separately?

[dsdashun]

Because in this commit pingcap/kvproto@f006036 , two fields are added into the S3 message . This PR will use the newly added fields to develop this feature. So I updated the kvproto version to include this commit.

[dsdashun]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 4820919

[sre-bot]

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test	✅ all 17 tests passed	14 min	Fixed
idc-jenkins-ci/integration-cdc-test	🟢 all 36 tests passed	25 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	13 min	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 13 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 53 sec	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	5 min 24 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 4 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 11 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 56 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

[ran-huang]

The new options should be documented in https://github.com/pingcap/docs-cn/blob/master/br/backup-and-restore-storages.md#s3-%E7%9A%84-url-%E5%8F%82%E6%95%B0. @dsdashun Can you please update the user doc?