tidb panic on br after inject pd leader io hang #47533
Labels
affects-6.5
affects-7.1
component/br
This issue is related to BR of TiDB.
severity/critical
type/bug
The issue is confirmed as a bug.
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1、run pitr
2、run workload
3、inject pd leader io hang
2. What did you expect to see? (Required)
no panic
3. What did you see instead (Required)
tidb panic
2023-10-11 06:57:28 log="\n"
2023-10-11 06:57:28 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-ticdc-tps-3000240-1-651.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n"
2023-10-11 06:57:28 log="start tidb-server ...\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:150 +0x78\n"
2023-10-11 06:57:27 log="created by github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable in goroutine 777\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:152 +0x4d\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable.func1()\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:234 +0x1f\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).doConnect.func1()\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:277 +0xbe5\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).listenOver(0xc020210b00, {0x60d80a0, 0xc04930c3b0})\n"
2023-10-11 06:57:27 log="goroutine 246752358 [running]:\n"
2023-10-11 06:57:27 log="\n"
2023-10-11 06:57:27 log="panic: send on closed channel\n"
2023-10-11 06:57:27 log="[flush_subscriber.go:144] ["join handle timed out."]"
2023-10-11 06:57:01 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:57:01 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:56:31 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:56:27 log="[flush_subscriber.go:297] ["Removing subscription."] [category="log backup subscription manager"] [store=4]"
2023-10-11 06:56:27 log="[flush_subscriber.go:144] ["join handle timed out."]"
4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v7.5.0-alpha
Edition: Community
Git Commit Hash: 97c7741
Git Branch: heads/refs/tags/v7.5.0-alpha
UTC Build Time: 2023-10-10 11:47:18
GoVersion: go1.21.1
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-10-11T06:39:11.534+0800
The text was updated successfully, but these errors were encountered: