Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFC] large transactions block watermark and result in replication lag increasing #10525

Closed
zhangjinpeng87 opened this issue Jan 23, 2024 · 0 comments · Fixed by #10526
Closed

[RFC] large transactions block watermark and result in replication lag increasing #10525

zhangjinpeng87 opened this issue Jan 23, 2024 · 0 comments · Fixed by #10526
Assignees
@zhangjinpeng87
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@zhangjinpeng87
Copy link
Contributor

zhangjinpeng87 commented Jan 23, 2024 •
edited
Loading

When there are large transactions running in upstream TiDB cluster, the watermark(resolve ts) will be blocked (explained in PR #10526), this will cause a large replication lag for TiCDC. We should have a good mechanism to smoothly advancing the watermark even there are some large transaction running in upstream TiDB cluster, to let TiCDC can replicate data in time and achieve better RPO.
@zhangjinpeng87 zhangjinpeng87 added the type/enhancement The issue or PR belongs to an enhancement. label Jan 23, 2024
@zhangjinpeng87 zhangjinpeng87 self-assigned this Jan 23, 2024
This was referenced Jan 23, 2024
Merged
Open
ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@zhangjinpeng87
[RFC] large transactions don't block watermark advancing (#10526)
d5ab31e 
close #10525
@zhangjinpeng87 zhangjinpeng87 changed the title large transactions block watermark and result in replication lag increasing [RFC] large transactions block watermark and result in replication lag increasing Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhangjinpeng87 zhangjinpeng87

Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[RFC] large transactions don't block watermark advancing zhangjinpeng87/tiflow
1 participant
@zhangjinpeng87