pkg/sql/schemachanger/schemachanger_test: TestConcurrentSchemaChanges failed [pq: at or near "EOF": syntax error]

[cockroach-teamcity]

pkg/sql/schemachanger/schemachanger_test.TestConcurrentSchemaChanges failed with artifacts on master @ 5d6498d8c4d279a7ba0ff058e1c2b0c1d22eb48c:

=== RUN   TestConcurrentSchemaChanges
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestConcurrentSchemaChanges2591203625
    test_log_scope.go:76: use -show-logs to present logs inline
    schemachanger_test.go:777: ADD COLUMN col_1 TO testdb.testsc.t
    schemachanger_test.go:781: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:781: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:781: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:781: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:781: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:728: DROP SCHEMA TO testsc
    schemachanger_test.go:728: DROP SCHEMA TO testsc
    schemachanger_test.go:726: RENAME SCHEMA TO testsc_9
    schemachanger_test.go:726: RENAME SCHEMA TO testsc_10
    schemachanger_test.go:777: ADD COLUMN col_14 TO testdb.testsc_10.t
    schemachanger_test.go:805: Parent database or schema or table is renamed; skipping this column removal.
    schemachanger_test.go:824: CREATE INDEX idx_15 ON testdb.testsc_10.t(col_14)
    schemachanger_test.go:824: CREATE INDEX idx_16 ON testdb.testsc_10.t(col_14)
    schemachanger_test.go:777: ADD COLUMN col_17 TO testdb.testsc_10.t
    schemachanger_test.go:777: ADD COLUMN col_18 TO testdb.testsc_10.t
    schemachanger_test.go:754: RENAME TABLE TO t_19
    schemachanger_test.go:673: add-column-worker encounters error pq: at or near "EOF": syntax error; signal to main routine and finish working
    schemachanger_test.go:669: rename-db-worker is signaled to finish work
    schemachanger_test.go:669: rename-tbl-worker is signaled to finish work
    schemachanger_test.go:669: drop-index-worker is signaled to finish work
    schemachanger_test.go:669: drop-column-worker is signaled to finish work
    schemachanger_test.go:669: rename-schema-worker is signaled to finish work
    schemachanger_test.go:669: create-index-worker is signaled to finish work
    schemachanger_test.go:863: 
        	Error Trace:	pkg/sql/schemachanger/schemachanger_test.go:863
        	Error:      	Received unexpected error:
        	            	pq: at or near "EOF": syntax error
        	Test:       	TestConcurrentSchemaChanges
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/2d9af299258c11d4d3ade298f6c5b40e/logTestConcurrentSchemaChanges2591203625
--- FAIL: TestConcurrentSchemaChanges (3.97s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-45653

[spilchen]

There is a parse error in the cockroach.log:

I241216 22:33:41.984155 2542 9@sql/pgwire/conn.go:385 ⋮ [T1,Vsystem,n1,client=127.0.0.1:44926,hostssl,user=root] 667  could not parse simple query: ‹ALTER TABLE testdb.testsc_10.t›

If this SQL were sent to the server, it would result in pq: at or near "EOF": syntax error. However, it's unclear how the test generated this truncated SQL. The failure occurs in the add-column-worker goroutine, which is responsible for generating SQL syntax. The relevant line is here:

cockroach/pkg/sql/schemachanger/schemachanger_test.go

Lines 774 to 775 in 2e55e6c

	_, err := workerConn.Exec(fmt.Sprintf("ALTER TABLE %v.%v.%v ADD COLUMN %v INT DEFAULT %v",
	dbName, scName, tblName, newColName, rand.Intn(100)))

It’s unclear how this goroutine could have produced an incomplete ALTER TABLE statement.

[spilchen]

Try to reproduce this under stress.

[cockroach-teamcity]

pkg/sql/schemachanger/schemachanger_test.TestConcurrentSchemaChanges failed on master @ 097438ac38e411b0fde101ebcae5cf97a798d1db:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestConcurrentSchemaChanges (15m0s)

Stack:

goroutine 439057 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

    schemachanger_test.go:803: DROP COLUMN col_206 FROM testdb_197.testsc_181.t_178
    schemachanger_test.go:756: RENAME TABLE TO t_211
    schemachanger_test.go:730: DROP SCHEMA TO testsc_181
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:694: DROP DATABASE testdb_197
    schemachanger_test.go:779: ADD COLUMN col_224 TO testdb_197.testsc_181.t_211
    schemachanger_test.go:826: CREATE INDEX idx_225 ON testdb_197.testsc_181.t_211(col_224)
    schemachanger_test.go:803: DROP COLUMN col_224 FROM testdb_197.testsc_181.t_211
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:756: RENAME TABLE TO t_226
    schemachanger_test.go:728: RENAME SCHEMA TO testsc_232
    schemachanger_test.go:779: ADD COLUMN col_236 TO testdb_197.testsc_232.t_226
    schemachanger_test.go:826: CREATE INDEX idx_237 ON testdb_197.testsc_232.t_226(col_236)
    schemachanger_test.go:803: DROP COLUMN col_236 FROM testdb_197.testsc_232.t_226
    schemachanger_test.go:694: DROP DATABASE testdb_197
    schemachanger_test.go:756: RENAME TABLE TO t_242
    schemachanger_test.go:779: ADD COLUMN col_238 TO testdb_197.testsc_232.t_226
    schemachanger_test.go:779: ADD COLUMN col_243 TO testdb_197.testsc_232.t_242
    schemachanger_test.go:779: ADD COLUMN col_244 TO testdb_197.testsc_232.t_242
    schemachanger_test.go:803: DROP COLUMN col_238 FROM testdb_197.testsc_232.t_242
    schemachanger_test.go:834: Parent database or schema or table is renamed or column is dropped or index already exists; skipping this index creation.
    schemachanger_test.go:826: CREATE INDEX idx_246 ON testdb_197.testsc_232.t_242(col_244)
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:728: RENAME SCHEMA TO testsc_248
    schemachanger_test.go:826: CREATE INDEX idx_247 ON testdb_197.testsc_232.t_242(col_243)
    schemachanger_test.go:834: Parent database or schema or table is renamed or column is dropped or index already exists; skipping this index creation.
    schemachanger_test.go:730: DROP SCHEMA TO testsc_248
    schemachanger_test.go:779: ADD COLUMN col_252 TO testdb_197.testsc_248.t_242
    schemachanger_test.go:803: DROP COLUMN col_252 FROM testdb_197.testsc_248.t_242
    schemachanger_test.go:783: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:834: Parent database or schema or table is renamed or column is dropped or index already exists; skipping this index creation.
    schemachanger_test.go:756: RENAME TABLE TO t_254
    schemachanger_test.go:705: RENAME DATABASE TO testdb_266
    schemachanger_test.go:705: RENAME DATABASE TO testdb_267
    schemachanger_test.go:779: ADD COLUMN col_260 TO testdb_197.testsc_248.t_254
    schemachanger_test.go:803: DROP COLUMN col_260 FROM testdb_267.testsc_248.t_254
    schemachanger_test.go:834: Parent database or schema or table is renamed or column is dropped or index already exists; skipping this index creation.
    schemachanger_test.go:730: DROP SCHEMA TO testsc_248
    schemachanger_test.go:728: RENAME SCHEMA TO testsc_271
    schemachanger_test.go:728: RENAME SCHEMA TO testsc_272
    schemachanger_test.go:671: drop-index-worker is signaled to finish work
    schemachanger_test.go:671: create-index-worker is signaled to finish work
    schemachanger_test.go:671: drop-column-worker is signaled to finish work

Parameters:

• attempt=1
• run=9
• shard=7

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/sql/schemachanger/schemachanger_test.TestConcurrentSchemaChanges failed on master @ 758fe6af0492e04f677196745f28a546b3a657cc:

Fatal error:

panic: test timed out after 15m0s
running tests:
	TestConcurrentSchemaChanges (15m0s)

Stack:

goroutine 547718 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

    schemachanger_test.go:749: RENAME TABLE TO t_729
    schemachanger_test.go:749: RENAME TABLE TO t_730
    schemachanger_test.go:749: RENAME TABLE TO t_731
    schemachanger_test.go:772: ADD COLUMN col_734 TO testdb_702.testsc_687.t_731
    schemachanger_test.go:796: DROP COLUMN col_734 FROM testdb_702.testsc_687.t_731
    schemachanger_test.go:772: ADD COLUMN col_737 TO testdb_702.testsc_687.t_731
    schemachanger_test.go:772: ADD COLUMN col_738 TO testdb_702.testsc_687.t_731
    schemachanger_test.go:796: DROP COLUMN col_738 FROM testdb_702.testsc_687.t_731
    schemachanger_test.go:723: DROP SCHEMA TO testsc_687
    schemachanger_test.go:772: ADD COLUMN col_741 TO testdb_702.testsc_687.t_731
    schemachanger_test.go:772: ADD COLUMN col_743 TO testdb_702.testsc_687.t_731
    schemachanger_test.go:796: DROP COLUMN col_741 FROM testdb_702.testsc_687.t_731
    schemachanger_test.go:827: Parent database or schema or table is renamed or column is dropped or index already exists; skipping this index creation.
    schemachanger_test.go:687: DROP DATABASE testdb_702
    schemachanger_test.go:751: DROP TABLE t_742
    schemachanger_test.go:749: RENAME TABLE TO t_745
    schemachanger_test.go:751: DROP TABLE t_746
    schemachanger_test.go:687: DROP DATABASE testdb_702
    schemachanger_test.go:772: ADD COLUMN col_747 TO testdb_702.testsc_687.t_745
    schemachanger_test.go:796: DROP COLUMN col_747 FROM testdb_702.testsc_687.t_745
    schemachanger_test.go:772: ADD COLUMN col_749 TO testdb_702.testsc_687.t_745
    schemachanger_test.go:796: DROP COLUMN col_749 FROM testdb_702.testsc_687.t_745
    schemachanger_test.go:723: DROP SCHEMA TO testsc_687
    schemachanger_test.go:721: RENAME SCHEMA TO testsc_753
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:723: DROP SCHEMA TO testsc_753
    schemachanger_test.go:756: Parent database or schema is renamed; skipping this table renaming.
    schemachanger_test.go:772: ADD COLUMN col_762 TO testdb_702.testsc_753.t_745
    schemachanger_test.go:796: DROP COLUMN col_762 FROM testdb_702.testsc_753.t_745
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:776: Parent database or schema or table is renamed or column already exists; skipping this column addition.
    schemachanger_test.go:698: RENAME DATABASE TO testdb_765
    schemachanger_test.go:698: RENAME DATABASE TO testdb_771
    schemachanger_test.go:772: ADD COLUMN col_775 TO testdb_771.testsc_753.t_745
    schemachanger_test.go:772: ADD COLUMN col_777 TO testdb_771.testsc_753.t_745
    schemachanger_test.go:796: DROP COLUMN col_775 FROM testdb_771.testsc_753.t_745
    schemachanger_test.go:796: DROP COLUMN col_777 FROM testdb_771.testsc_753.t_745
    schemachanger_test.go:772: ADD COLUMN col_779 TO testdb_771.testsc_753.t_745
    schemachanger_test.go:772: ADD COLUMN col_780 TO testdb_771.testsc_753.t_745
    schemachanger_test.go:664: rename-db-worker is signaled to finish work
    schemachanger_test.go:664: add-column-worker is signaled to finish work
    schemachanger_test.go:664: drop-index-worker is signaled to finish work
    schemachanger_test.go:796: DROP COLUMN col_779 FROM testdb_771.testsc_753.t_745
    schemachanger_test.go:664: drop-column-worker is signaled to finish work
    schemachanger_test.go:819: CREATE INDEX idx_781 ON testdb_771.testsc_753.t_745(col_780)
    schemachanger_test.go:664: create-index-worker is signaled to finish work

Parameters:

• attempt=1
• run=5
• shard=7

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[github-actions]

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.