meta/autoid: make autoid client ResetConn operation concurrency-safe

[tiancaiamao]

What problem does this PR solve?

Issue Number: close #50519

Problem Summary:

What changed and how does it work?

The old logic is not concurrency-safe. Although some of the fields are protected by mutex, the logic is not correct.

Imagine that there are 7K concurrent Alloc() operation, and one of them meet rpc error.
Then one ResetConn() is called, which invokes grpcConn.Close().
Note, there are many ongoing calling of Alloc() and still using the conn, so close the grpcConn cause this error:

rpc error: code = Canceled desc = grpc: the client connection is closing

This error in turn cause ResetConn() calling.
Even though GetClient() get a new client, the new client might be reset again by the ResetConn() mistakenly!
So the client can not recover from the error automatically some times (when the concurrent contention is high enough).

To summarize, there are too things we should avoid:

1. Calling grpcConn.Close() in ResetConn() when the grpcConn might still been using by some other sessions.
2. ResetConn() could be called multiple times and mistakenly reset the new connection.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Rename this to cmd/autoid/main.go and go run it (modify tidb/pkg/autoid_service/autoid.go and mock 1/1000 percent error rate also).

main.txt

Use the test described in #50519, after the fix, the QPS become stable:

go test -run TestXXX
==== qps ==== 120864
==== qps ==== 143747
==== qps ==== 143514
==== qps ==== 149449
==== qps ==== 144163
==== qps ==== 147491
==== qps ==== 139042
==== qps ==== 144311
==== qps ==== 146520
==== qps ==== 139134
==== qps ==== 144419
==== qps ==== 145943
==== qps ==== 146202
==== qps ==== 137703
==== qps ==== 142836
==== qps ==== 144951
==== qps ==== 142361
==== qps ==== 142149
==== qps ==== 148664
==== qps ==== 143449
==== qps ==== 146752
==== qps ==== 142496

And the error message like this is gone:

rpc error: code = Canceled desc = grpc: the client connection is closing

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Make autoid client ResetConn() operation concurrency-safe, get rid of error message like "rpc error: code = Canceled desc = grpc: the client connection is closing" when using AUTO_ID_CACHE=1

[tiprow]

Hi @tiancaiamao. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiancaiamao]

Introduce a version variable to avoid repeated ResetConn() and this also avoid a stale ResetConn() reset the newly created grpcConn mistakenly.

[tiancaiamao]

Delay the Close() operation, so the other session that still reference this grpcConn do not get the "grpc: the client connection is closing" error.

[codecov]

Codecov Report

Merging #50522 (d331238) into master (245951f) will decrease coverage by 16.1956%.
Report is 27 commits behind head on master.
The diff coverage is 16.0000%.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #50522         +/-   ##
=================================================
- Coverage   71.9131%   55.7175%   -16.1956%     
=================================================
  Files          1449       1564        +115     
  Lines        347105     588757     +241652     
=================================================
+ Hits         249614     328041      +78427     
- Misses        77205     237861     +160656     
- Partials      20286      22855       +2569     

Flag	Coverage Δ
integration	36.9161% <16.0000%> (?)
unit	70.3109% <16.0000%> (-1.6022%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	54.0269% <ø> (-2.2860%)	⬇️
parser	∅ <ø> (∅)
br	56.6142% <ø> (+5.3109%)	⬆️

[tiancaiamao]

/test mysql-test

[tiprow]

@tiancaiamao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiancaiamao]

/test mysql-test

[tiprow]

@tiancaiamao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bb7133]

LGTM

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-01-18 12:46:57.718512631 +0000 UTC m=+446459.282810326: ☑️ agreed by xhebox.
• 2024-01-19 02:21:23.031171427 +0000 UTC m=+495324.595469132: ☑️ agreed by bb7133.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, xhebox, zimulala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/meta/OWNERS [zimulala]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hawkingrei]

use atomic.Uint64 ?

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #50592.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #50605.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.6: #50621.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #51013.