Connection migration stops after a while

[salarali]

Bug Report

I have tidb's autoscaling. So coming up and down. After a while, the connection migration stops. I am assuming its because of tidb's coming up and down that breaks something. After restarting tiproxy, the connection migration resumes.

Let me know if you require more info for debugging.

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

Connection migration to continue working

3. What did you see instead (Required)

Connection migration to continue working

4. What is your version? (Required)

tiproxy version v1.3.2, commit 606669f
tidb version 8.5.1

[djshow832]

Firstly, check label_replace(tiproxy_balance_b_score{}, "backend", "$1", "backend", "(.+-tidb-[0-9]+).*peer.*.svc.*") to see the score of each backend.
High score means busy or unhealth. If the score of any backend is unexpected, we can check why that happens.

[salarali]

What I see is tidb instances that dont exist anymore, still in the list. Their values are static. 172.30.64.166 node doesnt exist anymore.

Could it be that because of this, it gets stuck somehow? There are several nodes like this.

[salarali]

This gives a better view of when it got stuck

[djshow832]

I guess 172.30.64.166 disappeared around 9:00?
status=1 means it's down and doesn't matter if it exists in metrics. The conn on it should be 0 after it disappears.

Check other TiDB instances please, especially those on which you think migration should happen.
status: 0 means alive, 1 means down
health: 0 means healthy, 1 means middle, 2 means unhealthy
memory: 0 means good, 1 means middle, 2 means OOM risk or high memory
cpu: [0, 20], higher value means higher CPU usage
location: 0 means local, 1 means remote
conn: the connection count on it

[salarali]

So this is another case here. Wasnt able to get it for above. As you can see, that status is 1 and conn is 1 as well.

Could this be the issue? Not all conns are moved over? This is for a dead instance.

[salarali]

Or could it be because of location. Somehow it gets set to remote?

[djshow832]

Could this be the issue? Not all conns are moved over? This is for a dead instance.

If you are sure that there won't be any connections on it, yes it should be an issue. If it's shutting down but the process still exists, maybe a long-running SQL is still on that connection and can't be migrated.

Or could it be because of location. Somehow it gets set to remote?

If the status is 1, all the connections must be migrated immediately, no matter what the location is.

[salarali]

Well the node doesn't exist anymore so it is definitely a problem on tiproxy side?

[djshow832]

Well the node doesn't exist anymore so it is definitely a problem on tiproxy side?

Yes. It will make the node always exist in the TiProxy memory, but I don't think it's a big problem.