[RFC] large transactions don't block watermark advancing

[zhangjinpeng87]

What problem does this PR solve?

Issue Number: close #10525

What is changed and how it works?

Propose a better watermark mechanism to advancing watermark smoothly when there are large transactions ongoing in upstream TiDB cluster.

Check List

Tests

• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`None`.

[codecov]

Codecov Report

Merging #10526 (b46c57b) into master (5d5ec56) will increase coverage by 0.0491%.
Report is 6 commits behind head on master.
The diff coverage is n/a.

❗ Current head b46c57b differs from pull request most recent head cdfef02. Consider uploading reports for the commit cdfef02 to get more accurate results

Additional details and impacted files

Components	Coverage Δ
cdc	61.8653% <ø> (+0.0767%)	⬆️
dm	51.2417% <ø> (+0.0162%)	⬆️
engine	63.4423% <ø> (+0.0494%)	⬆️

Flag	Coverage Δ
unit	57.5333% <ø> (+0.0491%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10526        +/-   ##
================================================
+ Coverage   57.4842%   57.5333%   +0.0491%     
================================================
  Files           849        849                
  Lines        126099     126099                
================================================
+ Hits          72487      72549        +62     
+ Misses        48195      48136        -59     
+ Partials       5417       5414         -3     

[ti-chi-bot]

@zhangjinpeng87: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cdc-integration-kafka-test	b46c57b	link	true	/test cdc-integration-kafka-test
pull-cdc-integration-storage-test	b46c57b	link	true	/test cdc-integration-storage-test
pull-cdc-integration-mysql-test	b46c57b	link	true	/test cdc-integration-mysql-test
pull-cdc-integration-pulsar-test	b46c57b	link	true	/test cdc-integration-pulsar-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CharlesCheung96, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CharlesCheung96,hicqu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-01-23 06:12:01.522858323 +0000 UTC m=+854763.087156023: ☑️ agreed by hicqu.
• 2024-01-24 03:52:43.362749886 +0000 UTC m=+932804.927047602: ☑️ agreed by CharlesCheung96.