roachtest: cdc/kafka-oauth failed

[cockroach-teamcity]

roachtest.cdc/kafka-oauth failed with artifacts on master @ a598a75435f6426236dde569191fbd95871f24ae:

(test_runner.go:1075).runTest: test timed out (3h0m0s)
(monitor.go:140).Wait: monitor failure: unexpected node event: n1: cockroach process died (exit code 137)
test artifacts and logs in: /artifacts/cdc/kafka-oauth/run_1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-30500

Epic CRDB-11732

[miretskiy]

It appears changefeed job was started, and completed successfully; yet the test continued waiting for changefeed to complete.

889949657184043009  CHANGEFEED  CREATE CHANGEFEED FOR TABLE tpcc.warehouse, TABLE tpcc.district, TABLE tpcc.customer, TABLE tpcc.history, TABLE tpcc.order, TABLE tpcc.new_order, TABLE tpcc.item, TABLE tpcc.stock, TABLE tpcc.order_line INTO 'kafka://10.142.1.82:9095?sasl_client_id=40447751-553b-457f-8693-24d05bda5ec9&sasl_client_secret=WmRsdTVYZUxKVWVsX1g3MnhieThuQkRtemg%3D&sasl_enabled=true&sasl_mechanism=OAUTHBEARER&sasl_token_url=http%3A%2F%2F10.142.1.82%3A4444%2Foauth2%2Ftoken' WITH OPTIONS (initial_scan = 'only', min_checkpoint_frequency = '10s')        root    {111,113,114,106,108,110,112,107,109}   succeeded   NULL    2023-08-10 09:58:01.90435+00    2023-08-10 09:58:01.942191+00   2023-08-10 09:58:31.3598+00 2023-08-10 09:58:31.359816+00   1   NULL        1   5382665569557280914 2023-08-10 09:58:01.942192+00   2023-08-10 09:58:31.942192+00   1   NULL    NULL

And it appears that the tester knew about that, but was stuck in the shutdown based on:

09:58:01 cdc.go:433: test status: created changefeed default with jobID 889949657184043009
09:58:31 cdc.go:315: test status: cdcTester closing
09:58:32 monitor.go:215: Monitor event: n2: cockroach process is running (PID: 11607)
09:58:32 monitor.go:215: Monitor event: n1: cockroach process is running (PID: 12303)
09:58:32 monitor.go:215: Monitor event: n3: cockroach process is running (PID: 12259)
12:55:19 test_impl.go:349: test failure #1: full stack retained in failure_1.log: (test_runner.go:1075).runTest: test timed out (3h0m0s)

And here is where the tester was stuck:

goroutine 1203736 [semacquire, 176 minutes]:
sync.runtime_Semacquire(0xc00400b140?)
    GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x5e2c540?)
    GOROOT/src/sync/waitgroup.go:139 +0x52
main.(*monitorImpl).wait(0xc0059411d0)
    main/pkg/cmd/roachtest/monitor.go:224 +0x1bb
main.(*monitorImpl).WaitE(0xc0059411d0)
    main/pkg/cmd/roachtest/monitor.go:128 +0x59
main.(*monitorImpl).Wait(0xc0059411d0)
    main/pkg/cmd/roachtest/monitor.go:136 +0x3f
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*cdcTester).Close(0xc004a7ab40)                                              github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:317 +0x8d
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func14({0x8490d88, 0xc006ef4780}, {0x851ec20, 0xc000f77a40}, {0x8553800, 0xc006035340})
    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1354 +0x5ba
main.(*testRunner).runTest.func2()
    main/pkg/cmd/roachtest/test_runner.go:1060 +0xa3
created by main.(*testRunner).runTest
    main/pkg/cmd/roachtest/test_runner.go:1045 +0x84c