Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data inconsistency seen when running storage sink and restarting PD/TiCDC/TiKV #9592

Closed
fubinzh opened this issue Aug 17, 2023 · 2 comments · Fixed by #9618
Closed

data inconsistency seen when running storage sink and restarting PD/TiCDC/TiKV #9592

fubinzh opened this issue Aug 17, 2023 · 2 comments · Fixed by #9618
Assignees
@CharlesCheung96
Labels
affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Aug 17, 2023

What did you do?

  1. Create storage sink changefeed
  2. Run workload
  3. Inject failure for all PD
  4. Inject failure for all TiCDC
  5. Inject failure for all TiKV
  6. Run storage consumer to consume storage sink content to downstream
  7. Compare data consistency between upstream and downstream

What did you expect to see?

Data should be consistent

What did you see instead?

Data inconsisteny seen

Versions of the cluster

CDC version:

[root@upstream-ticdc-0 /]# /cdc version
Release Version: v7.4.0-alpha
Git Commit Hash: dcfcb43a99bacd7639156421993a3a95284c5f45
Git Branch: heads/refs/tags/v7.4.0-alpha
UTC Build Time: 2023-08-16 11:36:28
Go Version: go version go1.21.0 linux/amd64
Failpoint Build: false
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Aug 17, 2023
@fubinzh
Copy link
Author

fubinzh commented Aug 17, 2023

/severity critical
/found automation

@CharlesCheung96 CharlesCheung96 self-assigned this Aug 17, 2023
@fubinzh
Copy link
Author

fubinzh commented Aug 21, 2023

/label affects-6.5
/label affects-7.1

@CharlesCheung96 CharlesCheung96 mentioned this issue Aug 21, 2023
Merged
@CharlesCheung96 CharlesCheung96 mentioned this issue Aug 21, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 22, 2023
@CharlesCheung96
sink(ticdc): cherry pick data loss fix to release 6.5 (#9621)
c0ac9fa 
close #9592
@CharlesCheung96 CharlesCheung96 mentioned this issue Aug 22, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 22, 2023
@CharlesCheung96
sink(ticdc): cherry pick data loss fix to hotfix release (#9623)
f39ba9f 
close #9592
ti-chi-bot bot pushed a commit that referenced this issue Aug 22, 2023
@CharlesCheung96
sink(ticdc): fix data loss (#9618)
0694352 
close #9592
This was referenced Aug 22, 2023
Closed
Merged
@nongfushanquan nongfushanquan mentioned this issue Aug 22, 2023
Merged
17 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 23, 2023
@ti-chi-bot
sink(ticdc): fix data loss (#9618) (#9628)
8196f8a 
close #9592
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Aug 29, 2023
@CharlesCheung96 @3AceShowHand
sink(ticdc): fix data loss (pingcap#9618)
62c07ee 
close pingcap#9592
@nongfushanquan nongfushanquan mentioned this issue Oct 18, 2023
Merged
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@CharlesCheung96 CharlesCheung96

Labels
affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. found/automation Bugs found by automation cases severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sink(ticdc): fix data loss CharlesCheung96/tiflow
2 participants
@fubinzh @CharlesCheung96