Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lease: keep alive Leadership immediately after finish the election of leader #7707

Closed
wants to merge 2 commits into from

Conversation

HuSharp
Copy link
Member

@HuSharp HuSharp commented Jan 15, 2024

What problem does this PR solve?

Issue Number: Ref #7251

What is changed and how does it work?

fix `etcdserver: requested lease not found`
// granted 3s
[2024/01/15 02:55:44.603 +00:00] [INFO] [server.go:1657] ["start to campaign PD leader"] [campaign-leader-name=tc-pd-0]
[2024/01/15 02:55:44.604 +00:00] [INFO] [lease.go:66] ["lease granted"] [lease-id=6177967866903092454] [lease-timeout=3] [purpose="leader election"]
// commit too slow
[2024/01/15 02:55:45.092 +00:00] [WARN] [etcdutil.go:152] ["kv gets too slow"] [request-key=/pd/7324144488088872732/member/5619078945150009647/leader_priority] [cost=2.764836133s] []
[2024/01/15 02:55:50.563 +00:00] [INFO] [etcdutil.go:314] ["update endpoints"] [num-change=3->2] [last-endpoints="[http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-1.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"] [endpoints="[http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"]
[2024/01/15 02:55:50.564 +00:00] [INFO] [etcdutil.go:314] ["update endpoints"] [num-change=3->2] [last-endpoints="[http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-1.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"] [endpoints="[http://tc-pd-2.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379,http://tc-pd-0.tc-pd-peer.endless-ha-test-tps-6030041-1-263.svc:2379]"]
[2024/01/15 02:55:51.020 +00:00] [WARN] [etcd_kv.go:174] ["txn runs too slow"] [response="{\"header\":{\"cluster_id\":10740517942856960270,\"member_id\":17574458819109638910,\"revision\":1502,\"raft_term\":16},\"succeeded\":true,\"responses\":[{\"Response\":{\"ResponsePut\":{\"header\":{\"revision\":1502}}}}]}"] [cost=6.415850764s] []
// `02:55:51` pass 7s than granted
[2024/01/15 02:55:51.020 +00:00] [INFO] [leadership.go:181] ["check campaign resp"] [resp="{\"header\":{\"cluster_id\":10740517942856960270,\"member_id\":17574458819109638910,\"revision\":1502,\"raft_term\":16},\"succeeded\":true,\"responses\":[{\"Response\":{\"ResponsePut\":{\"header\":{\"revision\":1502}}}}]}"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [leadership.go:190] ["write leaderData to leaderPath ok"] [leader-key=/pd/7324144488088872732/leader] [purpose="leader election"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [server.go:1683] ["campaign PD leader ok"] [campaign-leader-name=tc-pd-0]
[2024/01/15 02:55:51.020 +00:00] [INFO] [server.go:1691] ["initializing the global TSO allocator"]
[2024/01/15 02:55:51.020 +00:00] [INFO] [tso.go:160] ["start to sync timestamp"] []
// check lease failed
[2024/01/15 02:55:51.020 +00:00] [INFO] [lease.go:155] ["start lease keep alive worker"] [interval=1s] [purpose="leader election"]
[2024/01/15 02:55:51.020 +00:00] [WARN] [lease.go:170] ["lease keep alive failed"] [purpose="leader election"] [start=2024/01/15 02:55:51.020 +00:00] [error="etcdserver: requested lease not found"]
...
[2024/01/15 02:55:51.038 +00:00] [INFO] [lease.go:187] ["stop lease keep alive worker"] [purpose="leader election"]
[2024/01/15 02:55:51.039 +00:00] [INFO] [server.go:1657] ["start to campaign PD leader"] [campaign-leader-name=tc-pd-0]
...

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Release note

None.

Copy link
Contributor

ti-chi-bot bot commented Jan 15, 2024

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Copy link
Contributor

ti-chi-bot bot commented Jan 15, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Jan 15, 2024
@ti-chi-bot ti-chi-bot bot requested review from lhy1024 and rleungx January 15, 2024 07:24
@ti-chi-bot ti-chi-bot bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jan 15, 2024
@HuSharp HuSharp changed the title lease: regranted lease when put key path too long lease: regranted lease when write leader key too long Jan 15, 2024
Copy link

codecov bot commented Jan 15, 2024

Codecov Report

Merging #7707 (4292223) into master (cd0ffba) will decrease coverage by 0.18%.
Report is 1 commits behind head on master.
The diff coverage is 96.66%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7707      +/-   ##
==========================================
- Coverage   73.69%   73.51%   -0.18%     
==========================================
  Files         432      432              
  Lines       47805    47787      -18     
==========================================
- Hits        35229    35132      -97     
- Misses       9572     9637      +65     
- Partials     3004     3018      +14     
Flag Coverage Δ
unittests 73.51% <96.66%> (-0.18%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 23, 2024
@HuSharp HuSharp changed the title lease: regranted lease when write leader key too long lease: keep alive Leadership immediately after finish the election of leader Jan 23, 2024
Signed-off-by: husharp <jinhao.hu@pingcap.com>
@HuSharp HuSharp marked this pull request as ready for review January 23, 2024 04:30
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2024
@ti-chi-bot ti-chi-bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Feb 1, 2024
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. and removed needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels Feb 2, 2024
@lhy1024
Copy link
Contributor

lhy1024 commented May 20, 2024

Do we still need this pr? Will it need to be picked to 7.5?

@HuSharp
Copy link
Member Author

HuSharp commented May 20, 2024

Do we still need this pr? Will it need to be picked to 7.5?

I don't think this pr is needed for now, Let's close it for now.

@HuSharp HuSharp closed this May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants