auto analyze worker crashed when killed by tidb_mem_quota_analyze

[chrysan]

Bug Report

auto analyze worker panicked and did not recover when killed by global memory limit

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. set global memory limit to 6G
2. consume 5G long-existing memory
3. trigger an auto analyze who consumes 1G and appears as the top 1 memory consumer

2. What did you expect to see? (Required)

auto analyze got killed, but triggered next time

3. What did you see instead (Required)

auto analyze got killed, but never got triggered again

4. What is your TiDB version? (Required)

v6.5

[chrysan]

tidb-2022-12-12T15-55-52.981.log:[2022/12/12 07:09:59.607 +08:00] [ERROR] [misc.go:116] ["panic in the recoverable goroutine"] [label=domain] [funcInfo=autoAnalyzeWorker] [r="\"Out Of Global Analyze Memory Limit!\""] [stack="github.com/pingcap/tidb/util.Recover
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/misc.go:120
runtime.gopanic
\t/usr/local/go/src/runtime/panic.go:884
github.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:440
runtime.gopanic
\t/usr/local/go/src/runtime/panic.go:884
github.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:440
runtime.gopanic
\t/usr/local/go/src/runtime/panic.go:884
github.com/pingcap/tidb/executor.(*globalPanicOnExceed).Action
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:186
github.com/pingcap/tidb/util/memory.(*Tracker).Consume.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:437
github.com/pingcap/tidb/util/memory.(*Tracker).Consume
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/memory/tracker.go:478
github.com/pingcap/tidb/executor.(*UpdateExec).updateRows
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/update.go:274
github.com/pingcap/tidb/executor.(*UpdateExec).Next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/update.go:239
github.com/pingcap/tidb/executor.Next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328
github.com/pingcap/tidb/executor.(*ExecStmt).next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1142
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:915
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:763
github.com/pingcap/tidb/executor.(*ExecStmt).Exec
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:558
github.com/pingcap/tidb/session.runStmt
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2329
github.com/pingcap/tidb/session.(*session).ExecuteStmt
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2193
github.com/pingcap/tidb/session.(*session).ExecRestrictedSQL.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2075
github.com/pingcap/tidb/session.(*session).withRestrictedSQLExecutor
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2061
github.com/pingcap/tidb/session.(*session).ExecRestrictedSQL
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2065
github.com/pingcap/tidb/executor.FinishAnalyzeJob
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/analyze.go:554
github.com/pingcap/tidb/executor.finishJobWithLog
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/analyze.go:567
github.com/pingcap/tidb/executor.(*AnalyzeExec).handleResultsError
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/analyze.go:310
github.com/pingcap/tidb/executor.(*AnalyzeExec).Next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/analyze.go:185
github.com/pingcap/tidb/executor.Next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328
github.com/pingcap/tidb/executor.(*ExecStmt).next
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:1142
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:915
github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:763
github.com/pingcap/tidb/executor.(*ExecStmt).Exec
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:558
github.com/pingcap/tidb/session.runStmt
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2329
github.com/pingcap/tidb/session.(*session).ExecuteStmt
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2193
github.com/pingcap/tidb/session.(*session).ExecRestrictedSQL.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2075
github.com/pingcap/tidb/session.(*session).withRestrictedSQLExecutor
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2061
github.com/pingcap/tidb/session.(*session).ExecRestrictedSQL
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2065
github.com/pingcap/tidb/statistics/handle.(*Handle).execRestrictedSQLWithStatsVer.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:413
github.com/pingcap/tidb/statistics/handle.(*Handle).withRestrictedSQLExecutor
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:393
github.com/pingcap/tidb/statistics/handle.(*Handle).execRestrictedSQLWithStatsVer
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:405
github.com/pingcap/tidb/statistics/handle.(*Handle).execAutoAnalyze
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:1277
github.com/pingcap/tidb/statistics/handle.(*Handle).autoAnalyzeTable
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:1159
github.com/pingcap/tidb/statistics/handle.(*Handle).HandleAutoAnalyze
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/handle/update.go:1119
github.com/pingcap/tidb/domain.(*Domain).autoAnalyzeWorker
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:2019
github.com/pingcap/tidb/domain.(*Domain).UpdateTableStatsLoop.func2
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/domain/domain.go:1828
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1
\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33"]

[chrysan]

Since the panic happens in FinishAnalyzeJob, it's due to the session from sys session pool is not detached from GlobalAnalyzeMemTracker when returned back to pool and then used by internal queries in FinishAnalyzeJob.
Fix it in #39978