when TiKV returns rate limit errors, lightning should continue on next regions instead of retry on the busy region

[lance6716]

Feature Request

Is your feature request related to a problem? Please describe:

Currently lightning will recompute the region that need to ingest and retry from the start

tidb/br/pkg/lightning/backend/local/local.go

Line 1324 in 11f5c17

continue WriteAndIngest

when a region is busy, the logic will let lightning keeps retrying on this region.

Describe the feature you'd like:

lightning can continue on next regions

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[lance6716]

/cc @gozssky
do I understand the behaviour correctly?

[sleepymole]

Do you mean too many sst error? It actually means the node is busy. If continue, I think we should try regions that are not on this node.

[sleepymole]

Note that there is no check on the follower node. Before retrying, we'd better do some checks on the follower node.

[lance6716]

Do you mean too many sst error? It actually means the node is busy. If continue, I think we should try regions that are not on this node.

Yes currently too many sst is the only error of "rate limit" type, we might add more soon.

seems multi-rocksdb feature will be merged soon, I think we should not do that skipping-node optimization. On current single rocksdb architecture, skip only the busy region should improve some performance as well.

Note that there is no check on the follower node. Before retrying, we'd better do some checks on the follower node.

I'll consider merging #40116