Unexpected panic cased by index out of range in StatementContext.CopTasksDetails

[CbcWestwolf]

We found a panic in branch release-6.1-20221121-v6.1.1.

2022-12-18 14:03:34 | fatal error: sync: unlock of unlocked mutex
-- | --
2022-12-18 14:03:34 | [2022/12/18 06:03:34.900 +00:00] [WARN] [client_batch.go:794] ["wait response is cancelled"] [to=db-tikv-1.db-tikv-peer.tidbXXXXXXXXXX.svc:20160] [cause="context canceled"]
2022-12-18 14:03:34 |  
2022-12-18 14:03:34 | goroutine 674 [running]:
2022-12-18 14:03:34 | runtime.throw({0x3d49390?, 0x1731cde1aa842dae?})
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc057fe8eb0 sp=0xc057fe8e80 pc=0x148a091
2022-12-18 14:03:34 | sync.throw({0x3d49390?, 0x1731cde27e36fd14?})
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:978 +0x1e fp=0xc057fe8ed0 sp=0xc057fe8eb0 pc=0x14b8efe
2022-12-18 14:03:34 | sync.(*Mutex).unlockSlow(0xc03e141938, 0xffffffff)
2022-12-18 14:03:34 | 	/usr/local/go/src/sync/mutex.go:220 +0x3c fp=0xc057fe8ef8 sp=0xc057fe8ed0 pc=0x14cbdfc
2022-12-18 14:03:34 | sync.(*Mutex).Unlock(0x1731cde61cc3bd71?)
2022-12-18 14:03:34 | 	/usr/local/go/src/sync/mutex.go:214 +0x29 fp=0xc057fe8f18 sp=0xc057fe8ef8 pc=0x14cbd89
2022-12-18 14:03:34 | github.com/pingcap/tidb/sessionctx/stmtctx.(*StatementContext).CopTasksDetails.func4()
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/stmtctx/stmtctx.go:782 +0x26 fp=0xc057fe8f30 sp=0xc057fe8f18 pc=0x20b0b86
2022-12-18 14:03:34 | runtime.deferCallSave(0xc057fe9000, 0xc057fe9320?)
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:750 +0x82 fp=0xc057fe8f40 sp=0xc057fe8f30 pc=0x1489842
2022-12-18 14:03:34 | runtime.runOpenDeferFrame(0x10?, 0xc0250a0dc0)
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:723 +0x1a5 fp=0xc057fe8f88 sp=0xc057fe8f40 pc=0x1489665
2022-12-18 14:03:34 | panic({0x3affc20, 0xc006342810})
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:838 +0x207 fp=0xc057fe9048 sp=0xc057fe8f88 pc=0x1489aa7
2022-12-18 14:03:34 | runtime.goPanicIndex(0x0, 0x0)
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/panic.go:89 +0x7f fp=0xc057fe9088 sp=0xc057fe9048 pc=0x1487a1f
2022-12-18 14:03:34 | github.com/pingcap/tidb/sessionctx/stmtctx.(*StatementContext).CopTasksDetails.func1(0xc057fe9340?, 0xc0287e1580?)
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/stmtctx/stmtctx.go:800 +0x54 fp=0xc057fe90a8 sp=0xc057fe9088 pc=0x20b0b54
2022-12-18 14:03:34 | sort.doPivot_func({0xc057fe9340?, 0xc0287e1580?}, 0x0, 0x7b539)
2022-12-18 14:03:34 | 	/usr/local/go/src/sort/zfuncversion.go:87 +0x1d9 fp=0xc057fe9138 sp=0xc057fe90a8 pc=0x1528bb9
2022-12-18 14:03:34 | sort.quickSort_func({0xc057fe9340?, 0xc0287e1580?}, 0xc062e131d0?, 0x0?, 0x3b8f660?)
2022-12-18 14:03:34 | 	/usr/local/go/src/sort/zfuncversion.go:143 +0x85 fp=0xc057fe91a8 sp=0xc057fe9138 pc=0x1528f25
2022-12-18 14:03:34 | sort.Slice({0x3711b00, 0xc062e131d0}, 0x83c00?)
2022-12-18 14:03:34 | 	/usr/local/go/src/sort/slice.go:20 +0x9f fp=0xc057fe9208 sp=0xc057fe91a8 pc=0x152697f
2022-12-18 14:03:34 | github.com/pingcap/tidb/sessionctx/stmtctx.(*StatementContext).CopTasksDetails(0xc03e1418c8)
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/stmtctx/stmtctx.go:799 +0x28e fp=0xc057fe95c0 sp=0xc057fe9208 pc=0x20b018e
2022-12-18 14:03:34 | github.com/pingcap/tidb/util/expensivequery.genLogFields(0xdfe088928, 0xc026c6a620)
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/expensivequery/expensivequery.go:126 +0x2a5 fp=0xc057fe9bc0 sp=0xc057fe95c0 pc=0x2891c85
2022-12-18 14:03:34 | github.com/pingcap/tidb/util/expensivequery.logExpensiveQuery(0xc057fe9ed0?, 0x1?)
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/expensivequery/expensivequery.go:188 +0x70 fp=0xc057fe9c18 sp=0xc057fe9bc0 pc=0x2893090
2022-12-18 14:03:34 | github.com/pingcap/tidb/util/expensivequery.(*Handle).Run(0xc0008f0c18)
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/expensivequery/expensivequery.go:70 +0x38c fp=0xc057fe9fc8 sp=0xc057fe9c18 pc=0x2890f2c
2022-12-18 14:03:34 | main.createServer.func1()
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:730 +0x26 fp=0xc057fe9fe0 sp=0xc057fe9fc8 pc=0x35c8226
2022-12-18 14:03:34 | runtime.goexit()
2022-12-18 14:03:34 | 	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc057fe9fe8 sp=0xc057fe9fe0 pc=0x14bee01
2022-12-18 14:03:34 | created by main.createServer
2022-12-18 14:03:34 | 	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/tidb-server/main.go:730 +0x445

There are two unexpected behaviors in CopTasksDetails

1. Index Out of Range happened in sort.Slice()
2. Unlocked sc.mu is unlocked again, so it causes panic.

[CbcWestwolf]

/assign

[wshwsh12]

Same as #27725

[CbcWestwolf]

It has been fixed by #39368 in master. v5.3, v5.4 and v6.1 will be fixed by #39424, #39425, and #39426

[mjonss]

Lowered severity, since already fixed in master before 6.5 was frozen.

[CbcWestwolf]

We will close this issue since its root cause is the same as issue #27725