Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changefeed may stuck when tikv upgrade/restart/evict leader #10584

Closed
asddongmen opened this issue Feb 4, 2024 · 3 comments · Fixed by #10559
Closed

changefeed may stuck when tikv upgrade/restart/evict leader #10584

asddongmen opened this issue Feb 4, 2024 · 3 comments · Fixed by #10559
Assignees
@asddongmen
Labels
affects-5.4 affects-6.1 affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.

Comments

@asddongmen
Copy link
Contributor

asddongmen commented Feb 4, 2024 •
edited by kennytm
Loading

What did you do?

Rolling upgrade tikv.

What did you expect to see?

The changefeed checkpointTs lag less than 30s.
And more than thousand stream to store closed logs was found in cdc logs within 10 seconds.

What did you see instead?

The changefeed checkpointTs lag more than 10 minutes.
Additionally, more than a thousand stream to store closed logs were found in the cdc logs within 10 seconds.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

v6.5.7
@asddongmen asddongmen added type/bug The issue is confirmed as a bug. component/kv-client TiKV kv log client component. severity/major area/ticdc Issues or PRs related to TiCDC. labels Feb 4, 2024
@asddongmen asddongmen self-assigned this Feb 4, 2024
@asddongmen
Copy link
Contributor Author

ref: #10239 (comment)

This was referenced Feb 4, 2024
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 8, 2024
@asddongmen
kv (ticdc): Improve the codebase of kvClient. (#10585)
a609ffc 
ref #10584
@asddongmen
Copy link
Contributor Author

fix by #10559

@ti-chi-bot ti-chi-bot mentioned this issue Feb 18, 2024
Closed
@ti-chi-bot ti-chi-bot mentioned this issue Feb 20, 2024
Closed
@ti-chi-bot ti-chi-bot mentioned this issue Feb 29, 2024
Closed
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asddongmen asddongmen

Labels
affects-5.4 affects-6.1 affects-6.5 affects-7.1 area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.
Projects
Question and Bug Reports
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kv (ticdc): fix kvClient reconnection downhill loop asddongmen/tiflow
3 participants
@asddongmen @seiya-annie @ti-chi-bot