Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support service safe point (#2634) #2645

Merged

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #2634

What problem does this PR solve?

TiSpark does not have a mechanism to check GC. So, you may get the wrong data if your transaction's start_ts behind safe_point

What is changed and how it works?

This PR introduced the service safe point in TiSpark. TiKV will use the min service safe point among all service (including TiDB).

TiSpark will generate a new serviceId for every spark session. Every spark session will register the min start_ts as its' service safe point to PD. Thus, the safe point will not exceed the min start_ts among all spark applications.

Test

It is hard to write IT. Here are the results I test locally.

  1. PD logs show TiSpark register service safe point successfully

image

  1. TiKV metric shows TiSpark service safe point can change the safe point: TiSpark register 439743565061947392 (17:02:55) as safe point. This makes the GC safe point turns from 16:57:39 to 17:02:55 rather than 17:07:39

image

  1. TiKV metric shows TiSpark service safe point can pause the safe point.

image

  1. TiKV metric shows safe point will recovery after the TiSpark finished: The GC safe point turns from 17:02:55 to 17:18:39. which means TiSpark will not block it anymore.

image

  1. When the start_ts have already less than the current safe point. TiSpark will throw exceptions directly

3MUDaPC8Vq

  1. now-start_ts > spark.tispark.gc_max_wait_time will throw exception
    f94Xqc4Kqz

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member Author

ti-chi-bot commented Mar 10, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • shiyuhang0
  • zhangyangyu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

shiyuhang0
shiyuhang0 previously approved these changes Mar 11, 2023
@shiyuhang0
Copy link
Member

/run-all-tests

@shiyuhang0
Copy link
Member

/run-all-tests

@shiyuhang0
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member Author

This pull request has been accepted and is ready to merge.

Commit hash: 2c1cbc8

@ti-chi-bot ti-chi-bot merged commit cb779f0 into pingcap:release-3.0 Mar 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants