roachtest: disagg-rebalance/aws/n4cpu4 failed

[cockroach-teamcity]

roachtest.disagg-rebalance/aws/n4cpu4 failed with artifacts on master @ 54080dce12ba41c5578a1d1120b572198d884eaa:

(test_runner.go:1099).runTest: test timed out (1h0m0s)
(disagg_rebalance.go:92).func1: EOF
test artifacts and logs in: /artifacts/disagg-rebalance/aws/n4cpu4/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=aws , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-31123

[itsbilal]

Looks like n4 deadlocked on some rangefeed thing, causing waitForRebalance to time out? Seems pretty unrelated to what this test is testing.

[jbowens]

Looks like n4 deadlocked on some rangefeed thing, causing waitForRebalance to time out? Seems pretty unrelated to what this test is testing.

Even if it's unrelated to the test's intention, is there another team that should take a look and triage?

[itsbilal]

On closer inspection, the underlying issue is that n2 actually panicked:

I230901 06:05:30.231290 1 util/log/flags.go:213  [-] 1  stderr capture started
panic: runtime error: index out of range [33] with length 29

goroutine 20396 [running]:
panic({0x5e531e0, 0xc008e33530})
	GOROOT/src/runtime/panic.go:987 +0x3ba fp=0xc0036773a8 sp=0xc0036772e8 pc=0x498a9a
runtime.goPanicIndex(0x21, 0x1d)
	GOROOT/src/runtime/panic.go:113 +0x7f fp=0xc0036773e8 sp=0xc0036773a8 pc=0x4968bf
github.com/cockroachdb/pebble/internal/manifest.(*L0Sublevels).InitCompactingFileInfo(0xc00b82cfc0, {0xc0031182d0, 0x1, 0x1780b1594fc22029?})
	github.com/cockroachdb/pebble/internal/manifest/external/com_github_cockroachdb_pebble/internal/manifest/l0_sublevels.go:739 +0x4a9 fp=0xc003677668 sp=0xc0036773e8 pc=0x1244229
github.com/cockroachdb/pebble.(*DB).clearCompactingState(0xc00006a480, 0xc002cfaa80, 0x1?)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1698 +0x32a fp=0xc003677c58 sp=0xc003677668 pc=0x12f1eea
github.com/cockroachdb/pebble.(*DB).compact1(0xc00006a480, 0xc002cfaa80, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2671 +0x327 fp=0xc003677e68 sp=0xc003677c58 pc=0x12f8147
github.com/cockroachdb/pebble.(*DB).compact.func1({0x793ed90, 0xc000e98000})
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2579 +0xad fp=0xc003677ee8 sp=0xc003677e68 pc=0x12f7ced
runtime/pprof.Do({0x793ed20?, 0xc000208000?}, {{0xc0000c46a0?, 0x121dc80?, 0xc000b82b40?}}, 0xc0031c0f88)
	GOROOT/src/runtime/pprof/runtime.go:40 +0xa3 fp=0xc003677f58 sp=0xc003677ee8 pc=0xed7283
github.com/cockroachdb/pebble.(*DB).compact(0xc0031c0f90?, 0xc0031c0f60?, 0xc0031c0f70?)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2576 +0x6b fp=0xc003677fb8 sp=0xc003677f58 pc=0x12f7c0b
github.com/cockroachdb/pebble.(*DB).maybeScheduleCompactionPicker.func4()
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2297 +0x2c fp=0xc003677fe0 sp=0xc003677fb8 pc=0x12f616c
runtime.goexit()
	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc003677fe8 sp=0xc003677fe0 pc=0x4cdc21
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleCompactionPicker
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:2297 +0x62a

Seems like a new bug that we haven't seen before; I wonder if #109780 tickled it up in ways that we haven't seen it happen before. However lots of compactions were cancelled by excises on that node, and since we don't see compaction cancellations in non-disaggregated-storage clusters, it's probably not a serious bug for existing releases as it's exceedingly unlikely to trigger (But probably a bug regardless).

[jbowens]

Somehow related to cockroachdb/pebble#1781 and cockroachdb/pebble#1669?