crosscluster/physical: TestReplicationJobWithReaderTenant failed

[cockroach-teamcity]

crosscluster/physical.TestReplicationJobWithReaderTenant failed on release-26.1 @ c4f6da8bdbcd8d9d2e307998eeabc21f2b5946a9:

github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).releaseLoop(0xd5ee420)
	external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:61 +0xbf
created by github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).Init in goroutine 30286
	external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:54 +0x313

goroutine 34807 [select, 52 minutes, "cluster":"destination-readonly"]:
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3({0xd5ee420, 0xc004dc5a40})
	pkg/sql/virtual_table.go:118 +0x14d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd5ee420, 0xc004dc5a40})
	pkg/util/stop/stopper.go:440 +0x112
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 34772
	pkg/util/stop/stopper.go:438 +0x247

goroutine 34761 [sleep]:
time.Sleep(0x4c4b40)
	GOROOT/src/runtime/time.go:363 +0x165
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc007c66be0, {0xd5ee420, 0xc0086a80a0})
	pkg/util/stop/stopper.go:659 +0x173
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers.func1(0x0, 0xc007c66be0)
	pkg/testutils/testcluster/testcluster.go:174 +0x178
created by github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers in goroutine 19524
	pkg/testutils/testcluster/testcluster.go:170 +0x1c5

goroutine 34808 [select, "cluster":"destination-readonly"]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc00238d110)
	pkg/util/retry/retry.go:204 +0x555
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.storage.acquireBatch({0xc0086507c0, {0xd5ebf08, 0xc0059513b0}, 0xc0065058f0, 0xc008ef8000, {{0xc002281230, 0xc002281248, {0x3}}, {0xc002281230}, 0x0}, ...}, ...)
	pkg/sql/catalog/lease/storage.go:260 +0x7a5
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).EnsureBatch.func1()
	pkg/sql/catalog/lease/lease.go:1197 +0x7bf
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).EnsureBatch(0xc0058b9208, {0xd5ee420, 0xc004dc5c20}, {0x0, 0x0, 0x0})
	pkg/sql/catalog/lease/lease.go:1216 +0x2e5
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*leasedDescriptors).ensureLeasesExist(...)
	pkg/sql/catalog/descs/leased_descriptors.go:249
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetAllInDatabase(0xc0033bfc80, {0xd5ee420, 0xc004dc5c20}, 0xc004694dc0, {0xd6c5b60, 0xc0065c9a40}, {0xc00238dc38, 0x1, 0x1})
	pkg/sql/catalog/descs/collection.go:1157 +0x2c6
github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).GetAllDescriptorsForDatabase(0xc0033bfc80, {0xd5ee420, 0xc004dc5c20}, 0xc004694dc0, {0xd6c5b60, 0xc0059acf00}, {0xc004c6bc38, 0x1, 0x1})
	pkg/sql/catalog/descs/collection.go:1375 +0x185
github.com/cockroachdb/cockroach/pkg/sql.forEachTableDesc({0xd5ee420, 0xc004dc5c20}, 0xc00b1ded18, {0xd6c5b60, 0xc0059acf00}, {0x90000c004c6bc98?, 0xef?, 0xbc?}, 0xc00238dcd0)
	pkg/sql/information_schema.go:2762 +0x238
github.com/cockroachdb/cockroach/pkg/sql.init.func290.1({0xd5ee420, 0xc004dc5c20}, {0xd583300, 0xc0076a2bc0})
	pkg/sql/crdb_internal.go:3934 +0x214
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3.1(0xc001ca5780, {0xd5ee420, 0xc004dc5c20}, 0xc0076a2bc0)
	pkg/sql/virtual_table.go:129 +0xbb
github.com/cockroachdb/cockroach/pkg/sql.setupGenerator.func3({0xd5ee420, 0xc004dc5c20})
	pkg/sql/virtual_table.go:130 +0x17b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xd5ee420, 0xc004dc5c20})
	pkg/util/stop/stopper.go:440 +0x112
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 34772
	pkg/util/stop/stopper.go:438 +0x247

Parameters:

• attempt=1
• race=true
• run=2
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-58170

[dt]

Looks like something went wrong in scheduledlogging.captureIndexUsageStats; it has been running its crdb_internal query for 52 minutes, which in turn is what is preventing the node from shutting down cleanly until the test eventually timed out.