add index failed with error ““Error 1105 (HY000): rpc error: code = Canceled desc = context canceled” when inject one of tidb failure

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb_enable_dist_task='on'
1、run workload
2、add index for one table
3、inject one of tidb failure

case config
tag: "ha-test-add-index"
workloads:
- type: sysbench
storage: s3://benchmark/sysbench_64_7000w
db: sysbench_64_7000w
tableNum: 64
tableSize: 70000000
threads: 64
subType: oltp_read_write
ignoreErrors: "all"
cases:
- name: "ha_tidb_random_failure"
warmUpTime: 1m
faultType: failure
selector: "tidb(random)"
period: "@every 5m"
faultDuration: 3m
faultTotalRunTime: 30m
ddl:
addIndex:
- db: "sysbench_64_7000w"
table: "sbtest1"
timeout: 60m
times: 5
index_col_name: "c"
job_type: "ingest"
admin_check: false
count_check: false

2. What did you expect to see? (Required)

add index can success

3. What did you see instead (Required)

add index failed with error ““Error 1105 (HY000): rpc error: code = Canceled desc = context canceled” when inject one of tidb failure

add index failed at 2023-11-06 13:22:10 (Error 1105 (HY000): rpc error: code = Canceled desc = context canceled)
operatorLogs:
[2023-11-06 13:19:51] ###### start adding index
alter table sbtest1 add index index_test_1699247990976 (c)
[2023-11-06 13:19:51] ###### wait for ddl job finish

tidb log：
[2023/11/06 13:20:51.409 +08:00] [Error] [backfilling_operators.go:751] ["flush error"] [category=ddl] [error="peer 79078, store 4, region 79075, epoch conf_ver:29 version:1631 , when send data: rpc error: code = Canceled desc = context canceled"]
[2023/11/06 13:20:51.409 +08:00] [Error] [scheduler.go:487] [onError] [task-id=90001] [error="peer 79078, store 4, region 79075, epoch conf_ver:29 version:1631 , when send data: rpc error: code = Canceled desc = context canceled"] [stack="github.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).onError\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:487\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).runSubtask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:286\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:266\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*BaseScheduler).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/scheduler.go:135\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:385\ngithub.com/pingcap/tidb/pkg/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/scheduler/manager.go:217\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/spool/spool.go:144"]
[2023/11/06 13:20:51.409 +08:00] [Error] [scheduler.go:493] ["scheduler met first error"] [task-id=90001] [error="peer 79078, store 4, region 79075, epoch conf_ver:29 version:1631 , when send data: rpc error: code = Canceled desc = context canceled"]
[2023/11/06 13:20:51.409 +08:00] [Warn] [scheduler.go:614] ["subtask failed"] [task-id=90001] [error="peer 79078, store 4, region 79075, epoch conf_ver:29 version:1631 , when send data: rpc error: code = Canceled desc = context canceled"]

4. What is your TiDB version? (Required)

git hash：4f00ece106b1f9b13086d3e683939527d91718de

[Lily2025]

/assign ywqzzy