Skip to content

query cluster_slow_query table cause TiDB server oom #33893

@adunhappy

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1.进入dashboard慢查询界面;
2.选择业务高峰期的时间段,自定义时间范围略长一些,点击查询按钮;
3.到grafana中查看tidb-Overview命名空间下tidb组件的Memory Usage面板,发现内存直线上升后降低到0;
4.到以上第3步问题节点的服务器上,执行命令dmesg -T|grep tidb-server,能看到OOM的日志:
[Tue Apr 12 19:06:56 2022] tidb-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[Tue Apr 12 19:06:56 2022] tidb-server cpuset=/ mems_allowed=0
[Tue Apr 12 19:06:56 2022] CPU: 0 PID: 10434 Comm: tidb-server Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
[Tue Apr 12 19:06:56 2022] [10429] 1001 10429 290468 12488 101 0 0 tidb-server
[Tue Apr 12 19:06:56 2022] [17785] 1001 17785 2521970 1790373 4153 0 0 tidb-server
[Tue Apr 12 19:06:56 2022] Out of memory: Kill process 17785 (tidb-server) score 896 or sacrifice child
[Tue Apr 12 19:06:56 2022] Killed process 17785 (tidb-server), UID 1001, total-vm:10087880kB, anon-rss:7161492kB, file-rss:0kB, shmem-rss:0kB
[Tue Apr 12 19:08:55 2022] [10429] 1001 10429 290468 13355 101 0 0 tidb-server
[Tue Apr 12 19:08:55 2022] [17875] 1001 17875 2336392 1782316 4066 0 0 tidb-server
[Tue Apr 12 19:08:55 2022] Out of memory: Kill process 17875 (tidb-server) score 892 or sacrifice child
[Tue Apr 12 19:08:55 2022] Killed process 17875 (tidb-server), UID 1001, total-vm:9345568kB, anon-rss:7129264kB, file-rss:0kB, shmem-rss:0kB

2. What did you expect to see? (Required)

查询慢日志不会发生OOM

3. What did you see instead (Required)

查询慢日志发生OOM

4. What is your TiDB version? (Required)

Release Version: v5.1.2
Edition: Community
Git Commit Hash: dc07933
Git Branch: heads/refs/tags/v5.1.2
UTC Build Time: 2021-09-27 02:52:12
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

Metadata

Assignees

Labels

affects-4.0This bug affects 4.0.x versions.affects-5.0This bug affects 5.0.x versions.affects-5.1This bug affects 5.1.x versions.affects-5.2This bug affects 5.2.x versions.affects-5.3This bug affects 5.3.x versions.affects-5.4This bug affects the 5.4.x(LTS) versions.affects-6.0affects-6.1This bug affects the 6.1.x(LTS) versions.severity/majorsig/diagnosisSIG: Diagnosissig/executionSIG executiontype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions