Killing TiFlash causes TiDB down

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• Deploy a cluster with 1 tidb, 3 tikv, 2 tiflash, 1 pd.
• Run some workload ("schrodinger/bank2")
• Send kill -9 to a TiFlash instance
• The TiDB instance get restarted

2. What did you expect to see? (Required)

Killing TiFlash instance won't make TiDB get down

3. What did you see instead (Required)

Killing TiFlash instance DO make TiDB get down. The full logs:
tidb-log.tar.gz

>  cat log/tidb_stderr.log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2496f87]

goroutine 1342869 [running]:
github.com/pingcap/tidb/store/copr.balanceBatchCopTask.func1(0xc00105a760, 0xc00b60c6a0, 0x2, 0x2, 0xc00105a6f8, 0xc000672ea0, 0xc048e1b73280e987, 0x2daf1881f3, 0x5fadfa0, 0xc001136068, ...)
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:162 +0x3a7
created by github.com/pingcap/tidb/store/copr.balanceBatchCopTask
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:140 +0x16a5
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2496f87]

goroutine 36705 [running]:
github.com/pingcap/tidb/store/copr.balanceBatchCopTask.func1(0xc001dea260, 0xc002268d20, 0x2, 0x2, 0xc001dea258, 0xc006847a70, 0xc048e1bc3756cb4a, 0x11bf7b5d5, 0x5fadfa0, 0xc00059f4b0, ...)
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:162 +0x3a7
created by github.com/pingcap/tidb/store/copr.balanceBatchCopTask
	/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/src/github.com/pingcap/tidb/store/copr/batch_coprocessor.go:140 +0x16a5

4. What is your TiDB version? (Required)

mysql> select tidb_version();
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                                                                            |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v5.3.0-alpha-116-ge262e5984
Edition: Community
Git Commit Hash: e262e59842fbd0eb23f1da71448eaaf97581c4a8
Git Branch: master
UTC Build Time: 2021-09-15 07:55:38
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> select type,version,git_hash,start_time,uptime from information_schema.cluster_info;
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+
| type    | version                         | git_hash                                 | start_time                | uptime           |
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:30:00+08:00 | 7m5.451833153s   |
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:33:52+08:00 | 3m13.451838005s  |
| tidb    | 5.3.0-alpha                     | e262e59842fbd0eb23f1da71448eaaf97581c4a8 | 2021-09-16T17:33:32+08:00 | 3m33.451839836s  |
| pd      | 5.3.0-alpha                     | ef40841fee7f12c89c7562ae149013fb81a3e42b | 2021-09-16T16:46:25+08:00 | 50m40.451841512s |
| tiflash | 5.3.0-alpha-73-ge4b16122d-dirty | e4b16122d22b7664ca690b67a3d38f7e91a17369 | 2021-09-16T17:33:09+08:00 | 3m56.45184323s   |
| tiflash | 5.3.0-alpha-73-ge4b16122d-dirty | e4b16122d22b7664ca690b67a3d38f7e91a17369 | 2021-09-16T17:08:47+08:00 | 28m18.451846117s |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.45184788s  |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.451849806s |
| tikv    | 5.3.0-alpha                     | 6be3893f7f787b04bf34d99d1369092404ab5cfc | 2021-09-16T16:46:28+08:00 | 50m37.451852317s |
+---------+---------------------------------+------------------------------------------+---------------------------+------------------+

[JaySon-Huang]

I've tried to downgrade the TiDB server.
v5.2.0 has the same problem. But v5.1.1 does not.

[JaySon-Huang]

/assign @windtalker

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'backport-x.y.z',
and then remove 'needs-more-info' label.