Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pipeline: set table status to stopped after sink is closed #2716

Merged
merged 6 commits into from
Sep 2, 2021
Merged

pipeline: set table status to stopped after sink is closed #2716

merged 6 commits into from
Sep 2, 2021

Conversation

amyangfei
Copy link
Contributor

@amyangfei amyangfei commented Sep 1, 2021 •
edited
Loading

What problem does this PR solve?

Fix a data inconsistency issue caused by conflict write from two captures, it was a remaining bug after #2417

What is changed and how it works?

  • The table status of a pipeline should not set to stopped until the underlying sink is closed.
  • Remove unused StoppedTs parameter in pipeline command.

Check List

Tests

  • Unit test
  • Integration test

Release note 

Fix a bug that multiple processors could write the same table when this table is re-scheduling

@amyangfei amyangfei added subject/correctness Denotes an issue or pull request is related to correctness. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. labels Sep 1, 2021
@amyangfei amyangfei added this to the v5.3.0 milestone Sep 1, 2021
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 1, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • liuzix
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Sep 1, 2021
@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 1, 2021
@amyangfei
Copy link
Contributor Author

/run-all-tests

@codecov-commenter
Copy link

codecov-commenter commented Sep 1, 2021 •
edited
Loading

Codecov Report

Merging #2716 (f24f0fa) into master (81c22b1) will increase coverage by 5.6715%.
The diff coverage is 69.5798%.

@@               Coverage Diff                @@
##             master      #2716        +/-   ##
================================================
+ Coverage   55.8716%   61.5431%   +5.6715%     
================================================
  Files           169        161         -8     
  Lines         20667      17911      -2756     
================================================
- Hits          11547      11023       -524     
+ Misses         8012       5873      -2139     
+ Partials       1108       1015        -93

overvenus
overvenus approved these changes Sep 1, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

cdc/processor/processor.go Outdated
@@ -367,7 +367,7 @@ func (p *processor) handleTableOperation(ctx cdcContext.Context) error {
})
case model.OperProcessed:
if table.Status() != tablepipeline.TableStatusStopped {
log.Debug("the table is still not stopped", zap.Uint64("checkpointTs", table.CheckpointTs()), zap.Int64("tableID", tableID))
log.Info("the table is still not stopped", zap.Uint64("checkpointTs", table.CheckpointTs()), zap.Int64("tableID", tableID))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also change Debug to Info for the log at line 734?

https://github.com/pingcap/ticdc/blob/4f7c0b9606bc2c97f793caac42970aecc8f0a6c1/cdc/processor/processor.go#L734

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid there are too many logs if table count is large, such as 10k tables.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also revert this log level change, since

  1. we can deduce lazy stop from sink barrier block log
  2. It is better to refine scheduling logs in a specific PR

cdc/processor/pipeline/table.go Outdated
@@ -100,8 +100,7 @@ func (t *tablePipelineImpl) UpdateBarrierTs(ts model.Ts) {
// AsyncStop tells the pipeline to stop, and returns true is the pipeline is already stopped.
func (t *tablePipelineImpl) AsyncStop(targetTs model.Ts) bool {
err := t.p.SendToFirstNode(pipeline.CommandMessage(&pipeline.Command{
Tp: pipeline.CommandTypeStopAtTs,
StoppedTs: targetTs,
Tp: pipeline.CommandTypeStopAtTs,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Tp: pipeline.CommandTypeStopAtTs,
Tp: pipeline.CommandTypeStopAtTs,

Cloud you rename pipeline.CommandTypeStopAtTs to pipeline.CommandTypeStop ?

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 1, 2021
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 1, 2021
@amyangfei
Copy link
Contributor Author

/run-all-tests

1 similar comment
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/test ?

@amyangfei
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 4d40ea6

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 2, 2021
@amyangfei
Copy link
Contributor Author

/run-unit-tests
/run-kafka-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

2 similar comments
@amyangfei
Copy link
Contributor Author

/run-integration-tests

@amyangfei
Copy link
Contributor Author

/run-integration-tests

@amyangfei
Copy link
Contributor Author

/merge

@amyangfei
Copy link
Contributor Author

/run-integration-tests

@ti-chi-bot ti-chi-bot merged commit c97efbe into pingcap:master Sep 2, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Sep 2, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2727.

@ti-chi-bot ti-chi-bot mentioned this pull request Sep 2, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2728.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2729.

@ti-chi-bot ti-chi-bot mentioned this pull request Sep 2, 2021
Merged
@amyangfei amyangfei deleted the fix-sink-stop-state branch September 2, 2021 05:44
@ti-chi-bot ti-chi-bot mentioned this pull request Sep 2, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2730.

@nongfushanquan nongfushanquan added the cherry-pick-approved Cherry pick PR approved by release team. label Sep 6, 2021
ti-chi-bot added a commit that referenced this pull request Sep 6, 2021
@ti-chi-bot
pipeline: set table status to stopped after sink is closed (#2716) (#…
51b915e 
…2728)
ti-chi-bot added a commit that referenced this pull request Sep 6, 2021
@ti-chi-bot
pipeline: set table status to stopped after sink is closed (#2716) (#…
77da864 
…2727)
ti-chi-bot added a commit that referenced this pull request Sep 6, 2021
@ti-chi-bot
pipeline: set table status to stopped after sink is closed (#2716) (#…
5e43787 
…2730)
ti-chi-bot added a commit that referenced this pull request Sep 6, 2021
@ti-chi-bot
pipeline: set table status to stopped after sink is closed (#2716) (#…
7bbd3ee 
…2729)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus approved these changes

@liuzix liuzix liuzix approved these changes

@ben1009 ben1009 Awaiting requested review from ben1009

Assignees
No one assigned
Labels
cherry-pick-approved Cherry pick PR approved by release team. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. subject/correctness Denotes an issue or pull request is related to correctness.
Projects
None yet
Milestone
v5.3.0
Development

Successfully merging this pull request may close these issues.
6 participants
@amyangfei @ti-chi-bot @codecov-commenter @overvenus @liuzix @nongfushanquan