add index or lightning or import into failed with error “Error 1105 (HY000): request pd http api failed with status: '503 Service Unavailable” when kill pd leader

[Lily2025]

Bug Report

What did you do?

1、add index
2、kill pd leader

What did you expect to see?

add index can success

What did you see instead?

add index failed with error “Error 1105 (HY000): request pd http api failed with status: '503 Service Unavailable”

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v8.2.0-alpha
Edition: Community
Git Commit Hash: 1679dbc
Git Branch: heads/refs/tags/v8.2.0-alpha
UTC Build Time: 2024-04-30 11:39:01

[Lily2025]

/type bug
/severity major
/assign rleungx

[Lily2025]

/assign JmPotato

[Lily2025]

lightning failed with error “request pd http api failed with status: '500 Internal Server Error'“ when kill pd leader

lightning logs：
[2024/05/06 15:42:15.072 +00:00] [ERROR] [client.go:234] ["[pd] request failed with a non-200 status"] [source=lightning] [name=GetStores] [url=http://tc-pd-2.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/stores] [method=GET] [caller-id=pd-http-client] [status="500 Internal Server Error"] [body="[PD:apiutil:ErrRedirect]redirect failed\n"] [stack="github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:234\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:139\ngithub.com/tikv/pd/client/retry.(*Backoffer).Exec\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/retry/backoff.go:78\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:160\ngithub.com/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:379\ngithub.com/tikv/pd/client/http.(*client).GetStores\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/interface.go:403\ngithub.com/pingcap/tidb/pkg/lightning/tikv.ForAllStores\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/tikv/tikv.go:103\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*switcher).switchTiKVMode\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/tikv_mode.go:69\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*switcher).ToImportMode\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/tikv_mode.go:53\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*Controller).buildRunPeriodicActionAndCancelFunc.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/import.go:1001"] [2024/05/06 15:42:16.034 +00:00] [ERROR] [client.go:206] ["[pd] do http request failed"] [source=lightning] [name=SetRegionLabelRule] [url=http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/config/region-label/rule] [method=POST] [caller-id=pd-http-client] [error="Post \"http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7575769-1-386.svc:2379/pd/api/v1/config/region-label/rule\": dial tcp 10.200.72.140:2379: connect: connection refused"] [stack="github.com/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:206\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry.func1\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:139\ngithub.com/tikv/pd/client/retry.(*Backoffer).Exec\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/retry/backoff.go:78\ngithub.com/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:160\ngithub.com/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/client.go:379\ngithub.com/tikv/pd/client/http.(*client).SetRegionLabelRule\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20240430080403-1679dbca25b3/http/interface.go:691\ngithub.com/pingcap/tidb/br/pkg/pdutil.pauseSchedulerByKeyRangeWithTTL\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:702\ngithub.com/pingcap/tidb/br/pkg/pdutil.PauseSchedulersByKeyRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:670\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).ImportEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/local/local.go:1291\ngithub.com/pingcap/tidb/pkg/lightning/backend.(*ClosedEngine).Import\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/lightning/backend/backend.go:373\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importKV\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:1346\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:920\ngithub.com/pingcap/tidb/lightning/pkg/importer.(*TableImporter).importEngines.func3\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/lightning/pkg/importer/table_import.go:526"] [2024/05/06 15:42:16.060 +00:00] [ERROR] [backend.go:378] ["import failed"]

[JmPotato]

Introduced by #7896, multi-errors will return an error even if the request eventually succeeds after retries.