kv/client: send resovled ts in advance when region is locked (#2038)

[ti-chi-bot]

This is an automated cherry-pick of #2038

What problem does this PR solve?

A workaround fix for #1576

Since the region can merge and split at any given time, we have found one possible scenario that can cause panic in #1576, as following

• A, B, C, D, E are region range key.
• In the first line each range represents a region range, such as [A, B) represents range from A to B.
• The second line represents the resolved-ts maintained in puller frontier.

1. 
[A, B) [B, C) [C, D) [D, E)
20     20     10     20

2. [B, C) [C, D) merged into [B, D). But [B, D) does not forward, because of split to [B, C) and [C, D) immediately
[A, B) [B, D) [D, E)
20     10     20

3. [B, D) split to [B, C) and [C, D). [B, C) does not forward, because of merge into [A, B) immediately
[A, B) [B, C) [C, D) [D, E)
20     10     10     20

4.  Bump, here we meet resolved ts fall back, where 10 is less than the resolved ts (20) in puller. But there is no data loss actually.
region [A, C) starts from resolved-ts=10

What is changed and how it works?

• Send a resolved ts to event channel first, for two reasons:
  1. Since we have locked the region range, and have maintained correct
     checkpoint ts for the range, it is safe to report the resolved ts
     to puller at this moment.
  2. Before the kv client gets region rpcCtx, sends request to TiKV and
     receives the first kv event from TiKV, the region could split or
     merge in advance, which should cause the change of resolved ts
     distribution in puller, so this resolved ts event is needed.

Check List

Tests

• Unit test
• Integration test

Release note

• Fix a bug that could cause cdc server panic because of the late calculation of resolved ts

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

@ti-chi-bot: This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick, it must first be approved by the collaborators.

AFTER it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[amyangfei]

Cherry-picked in #2149