roachtest: splits/largerange/size=32GiB,nodes=6 failed [OOM]

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 6723e00a46aaa3ea575093bd82a02b7d6f6b131b:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(test_impl.go:291).Fatal: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-22859

Epic CRDB-18656

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 64e4fc9faa4e0ab19fe5ba78f053bc2b1390cb5e:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 250238cd29102391dddbc8cc71380090c49ce509:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(sql_runner.go:194).Query: error executing '
select concat('r', range_id::string) as range, voting_replicas
from [ SHOW RANGES FROM DATABASE bank ] where cardinality(voting_replicas) >= $1;': read tcp 172.17.0.3:59572 -> 34.23.31.153:26257: read: connection reset by peer
(monitor.go:127).Wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 1: dead (exit status 137)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.splits/largerange/size=32GiB,nodes=6 failed with artifacts on master @ 2c3d75f1ce31024d7ffe530f91f22162c053abcd:

test artifacts and logs in: /artifacts/splits/largerange/size=32GiB_nodes=6/run_1
(sql_runner.go:194).Query: error executing '
select concat('r', range_id::string) as range, voting_replicas
from [ SHOW RANGES FROM DATABASE bank ] where cardinality(voting_replicas) >= $1;': dial tcp 34.148.222.162:26257: connect: connection refused
(monitor.go:127).Wait: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[erikgrinaker]

n1 OOM-killed. Pretty sure this is fallout from #92858, which also caused OOMs in #93579, and is possibly fixed by #93823. Reassigning to @kvoli to confirm.

[ 8137.222561] cockroach invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[ 8137.222566] CPU: 1 PID: 39023 Comm: cockroach Not tainted 5.4.0-1044-gcp #47-Ubuntu
[ 8137.222568] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
[ 8137.222569] Call Trace:
[ 8137.222579]  dump_stack+0x6d/0x8b
[ 8137.222583]  dump_header+0x4f/0x1eb
[ 8137.222584]  oom_kill_process.cold+0xb/0x10
[ 8137.222588]  out_of_memory.part.0+0x1df/0x3d0
[ 8137.222589]  out_of_memory+0x6d/0xd0
[ 8137.222592]  mem_cgroup_out_of_memory+0xbd/0xe0
[ 8137.222595]  try_charge+0x77c/0x810
[ 8137.222597]  mem_cgroup_try_charge+0x71/0x190
[ 8137.222599]  __add_to_page_cache_locked+0x265/0x340
[ 8137.222602]  ? scan_shadow_nodes+0x30/0x30
[ 8137.222603]  add_to_page_cache_lru+0x4d/0xd0
[ 8137.222604]  pagecache_get_page+0x101/0x300
[ 8137.222606]  filemap_fault+0x6b2/0xa50
[ 8137.222607]  ? unlock_page_memcg+0x12/0x20
[ 8137.222610]  ? page_add_file_rmap+0xff/0x1a0
[ 8137.222611]  ? xas_load+0xd/0x80
[ 8137.222612]  ? xas_find+0x17f/0x1c0
[ 8137.222614]  ? filemap_map_pages+0x24c/0x380
[ 8137.222617]  ext4_filemap_fault+0x32/0x50
[ 8137.222618]  __do_fault+0x3c/0x130
[ 8137.222620]  do_fault+0x24b/0x640
[ 8137.222622]  __handle_mm_fault+0x4c5/0x7a0
[ 8137.222624]  handle_mm_fault+0xca/0x200
[ 8137.222627]  do_user_addr_fault+0x1f9/0x450
[ 8137.222628]  __do_page_fault+0x58/0x90
[ 8137.222629]  do_page_fault+0x2c/0xe0
[ 8137.222633]  page_fault+0x34/0x40
[ 8137.222635] RIP: 0033:0x67c52e
[ 8137.222637] Code: c0 4d 3b 66 10 0f 86 d1 0d 00 00 48 81 ec c0 00 00 00 48 89 ac 24 b8 00 00 00 48 8d ac 24 b8 00 00 00 48 89 84 24 c8 00 00 00 <4c> 8b 0d 63 3a 2c 09 4c 8b 15 54 3a 2c 09 4c 8b 1d f5 39 2c 09 4c
[ 8137.222638] RSP: 002b:000000c005893230 EFLAGS: 00010206
[ 8137.222639] RAX: 000000c07f6663c0 RBX: 0000000000000007 RCX: 0000000000000040
[ 8137.222640] RDX: 000000c07f6663c0 RSI: 0000000000000000 RDI: 0000000000000000
[ 8137.222641] RBP: 000000c0058932e8 R08: 0000000000000000 R09: 0000000000000000
[ 8137.222642] R10: 00007f0e0cbad3d8 R11: 0000000000000040 R12: 000000c0058932b0
[ 8137.222642] R13: 0000000000000000 R14: 000000c0060b49c0 R15: ffffffffffffffff
[ 8137.222645] memory: usage 14594444kB, limit 14594444kB, failcnt 4617261
[ 8137.222645] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[ 8137.222646] kmem: usage 51880kB, limit 9007199254740988kB, failcnt 0
[ 8137.222647] Memory cgroup stats for /system.slice/cockroach.service:
[ 8137.222657] anon 14890971136
               file 253952
               kernel_stack 184320
               slab 21319680
               sock 0
               shmem 0
               file_mapped 0
               file_dirty 135168
               file_writeback 270336
               anon_thp 117440512
               inactive_anon 0
               active_anon 14890999808
               inactive_file 0
               active_file 217088
               unevictable 0
               slab_reclaimable 20283392
               slab_unreclaimable 1036288
               pgfault 20381229
               pgmajfault 990
               workingset_refault 2023098
               workingset_activate 49962
               workingset_nodereclaim 0
               pgrefill 18433
               pgscan 8838721
               pgsteal 8814947
               pgactivate 12408
               pgdeactivate 16641
               pglazyfree 0
               pglazyfreed 0
               thp_fault_alloc 66
               thp_collapse_alloc 0
[ 8137.222658] Tasks state (memory values in pages):
[ 8137.222658] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 8137.222661] [  38401]  1000 38401     2154      618    57344        0             0 bash
[ 8137.222662] [  38406]  1000 38406  4099404  3643144 31518720        0             0 cockroach
[ 8137.222664] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/cockroach.service,task_memcg=/system.slice/cockroach.service,task=cockroach,pid=38406,uid=1000
[ 8137.222716] Memory cgroup out of memory: Killed process 38406 (cockroach) total-vm:16397616kB, anon-rss:14540508kB, file-rss:32068kB, shmem-rss:0kB, UID:1000 pgtables:30780kB oom_score_adj:0
[ 8137.992502] oom_reaper: reaped process 38406 (cockroach), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB