kv (ticdc): fix kvClient reconnection downhill loop (#10559)

[ti-chi-bot]

This is an automated cherry-pick of #10559

What problem does this PR solve?

Issue Number: close #10239

What is changed and how it works?

1. Add an ID to the eventFeedStream struct to identify a stream and prevent it from being deleted unexpectedly.
2. Bind the cancel function of a gRPC stream to its eventFeedStream to prevent the stream from being canceled unexpectedly.
3. Reduce the number of calls to s.deleteStream to only once to prevent the stream from being deleted unexpectedly.

Check List

Tests

• Manual test (add detailed scripts or steps below)

1. Add hard code time.sleep(5 * time.second) before calling s.deleteStream in both unfixed and fixed version of code.
2. Deploy a TiDB cluster with 3 TiKV nodes and 1 TiCDC server, create a changefeed.
3. Patch the changed cdc binary, and restart a TiKV node randomly and observe the changefeed's lag.

unfixed cdc:

fixed cdc:

From the above graphs, it is evident that in the unfixed CDC, the lag of resolvedTs can exceed 12 minutes when a TiKV node is restarted. However, in the fixed CDC, the increase in resolvedTs is limited to a maximum of 35 seconds. This demonstrates the effectiveness of the fix.

Moreover, when the hard-coded time.sleep(5 * time.second) is removed and the fixed version of CDC is tested again, the lag becomes even smaller:

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug in kv client that could cause an increase in changefeed lag when TiKV is restarted.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gozssky for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
jenkins-ticdc/verify	fc9ba7e	link	true	/test verify
pull-cdc-integration-storage-test	fc9ba7e	link	true	/test cdc-integration-storage-test
pull-cdc-integration-mysql-test	fc9ba7e	link	true	/test cdc-integration-mysql-test
pull-cdc-integration-pulsar-test	fc9ba7e	link	true	/test cdc-integration-pulsar-test
pull-cdc-integration-kafka-test	fc9ba7e	link	true	/test cdc-integration-kafka-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[asddongmen]

This issue can not happen in 7.5.0, so close it.