roachprod-microbench: panic due to non-crdb-test build

[herkolategan]

Describe the problem

[08:58:26 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1502&logView=flowAware&linesState=1502)  + ./bin/roachprod-microbench run teamcity-microbench-20059279 --binaries experiment=/mnt/data1/20aa5054030c4e802024e734ea588db21257ab3a --output-dir=./artifacts/microbench --iterations 10 --shell=COCKROACH_RANDOM_SEED=1 --timeout=20m '--exclude=.*:BenchmarkDatabaseBackup,.*:BenchmarkDatabaseRestore,.*:BenchmarkIntentRangeResolution,.*:BenchmarkMVCCGarbageCollect,.*:BenchmarkMVCCReverseScan_Pebble,.*:BenchmarkMVCCScanGarbage_Pebble,.*:BenchmarkORMQueries,.*:BenchmarkLockTable,.*:BenchmarkLockTableMetrics' --ignore-package=pkg/testutils/lint,pkg/sql/schemachanger/scbuild --post-issues --quiet -- -test.short
[08:58:27 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1503&logView=flowAware&linesState=1503)  teamcity-microbench-20059279: status
[08:58:27 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1504&logView=flowAware&linesState=1504)  2025/06/29 08:58:27 executor.go:371: number of nodes to execute benchmarks on: 12
[08:58:32 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1505&logView=flowAware&linesState=1505)  2025/06/29 08:58:32 executor.go:218: Distributing and running benchmark listings across cluster teamcity-microbench-20059279
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1506&logView=flowAware&linesState=1506)  2025/06/29 08:58:51 executor.go:213: Remote command = {sh -c "[ ! -d /mnt/data1/20aa5054030c4e802024e734ea588db21257ab3a/pkg/sql/schemachanger/scplan/bin ] || (cd /mnt/data1/20aa5054030c4e802024e734ea588db21257ab3a/pkg/sql/schemachanger/scplan/bin && ./run.sh -test.list=^Benchmark*)"}, error = {COMMAND_PROBLEM: exit status 134}, stderr output = {panic: Testing override for schema_locked used in non-test binary.
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1507&logView=flowAware&linesState=1507)  
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1508&logView=flowAware&linesState=1508)  goroutine 1 gp=0xc0000061c0 m=0 mp=0xcdc13e0 [running]:
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1509&logView=flowAware&linesState=1509)  panic({0x660dec0?, 0x830d430?})
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1510&logView=flowAware&linesState=1510)    GOROOT/src/runtime/panic.go:810 +0x168 fp=0xc003207e00 sp=0xc003207d50 pc=0x4d4748
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1511&logView=flowAware&linesState=1511)  github.com/cockroachdb/cockroach/pkg/sql.TestForceDisableCreateTableWithSchemaLocked(...)
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1512&logView=flowAware&linesState=1512)    pkg/sql/exec_util.go:773
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1513&logView=flowAware&linesState=1513)  pkg/sql/schemachanger/scplan/scplan_test_test.init.0()
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1514&logView=flowAware&linesState=1514)    pkg/sql/schemachanger/scplan/plan_test.go:46 +0x25 fp=0xc003207e20 sp=0xc003207e00 pc=0x5f0b365
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1515&logView=flowAware&linesState=1515)  runtime.doInit1(0xca23390)
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1516&logView=flowAware&linesState=1516)    GOROOT/src/runtime/proc.go:7321 +0xe8 fp=0xc003207f50 sp=0xc003207e20 pc=0x4ac608
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1517&logView=flowAware&linesState=1517)  runtime.doInit(...)
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1518&logView=flowAware&linesState=1518)    GOROOT/src/runtime/proc.go:7288
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1519&logView=flowAware&linesState=1519)  runtime.main()
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1520&logView=flowAware&linesState=1520)    GOROOT/src/runtime/proc.go:254 +0x345 fp=0xc003207fe0 sp=0xc003207f50 pc=0x49dd65
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1521&logView=flowAware&linesState=1521)  runtime.goexit({})
[08:58:51 ](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachprodMicrobenchmarkWeeklyWip/20059279?buildTab=log&focusLine=1522&logView=flowAware&linesState=1522)    src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003207fe8 sp=0xc003207fe0 pc=0x4ddcc1

To Reproduce

Build a test binary with the crdb_test flag.

The scplan package fails on init due to this panic.

These are for the Microbenchmark Weekly Runs.

Ons solution is to maybe add crdb_test to the build options for the Microbenchmark Weekly build step, but this could possibly have performance side effects, or skew comparisons.

Jira issue: CRDB-52105

[blathers-crl]

Hi @herkolategan, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/test-eng

[srosenberg]

Ons solution is to maybe add crdb_test to the build options for the Microbenchmark Weekly build step, but this could possibly have performance side effects, or skew comparisons.

crdb_test is a can of worms for microbenches. Aside from the unwanted heavy checks, the binary will miss some inlining and other optimizations due to heap escapes.

To avoid the panic, we basically need to confirm it's a test binary. Short of defining a new build tag, we could simply check for a runfile (via bazel.Runfile), assuming the test package has it. Luckily, pkg/sql has testdata.

[blathers-crl]

Based on the specified backports for linked PR #149517, I applied the following new label(s) to this issue: branch-release-25.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}