pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [ALTER PK failed validation: expected 0 rows, found 4]

[cockroach-teamcity]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on master @ bb1c098ecc7058a2b24a23820c5a8e235dfa714c:

     true,
     true,
     true,
     true,
     true,
     true,
     true,
     false
    ]
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w0_141 ALTER PRIMARY KEY USING COLUMNS (col141_w0_147, col141_w0_149, col141_w0_152) USING HASH"
  ]
 }
}
    schema_change_external_test.go:165: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:165
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (XX000): executing declarative schema change PostCommitPhase stage 7 of 16 with 3 ValidationType ops (rollback=false) for ALTER TABLE: validation of index crdb_internal_index_4_name_placeholder failed: expected 0 rows, found 4 (SQLSTATE XX000)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	pkg/workload/schemachange/schemachange.go:682
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func5.func7
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:154
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1223
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (XX000): executing declarative schema change PostCommitPhase stage 7 of 16 with 3 ValidationType ops (rollback=false) for ALTER TABLE: validation of index crdb_internal_index_4_name_placeholder failed: expected 0 rows, found 4 (SQLSTATE XX000)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 84
Total Schema Statements Succeeded = 84
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 57
Total Transactions Rolled Back = 40
Total Transactions Executed = 97
    schema_change_external_test.go:118: backup, tracing data, and system table dumps in /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload4294111833
    schema_change_external_test.go:127: running DROP with use_declarative_schema_changer = on
    panic.go:635: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/67537770b84c3a4ea177539c1e46bd43/logTestWorkload4294111833
--- FAIL: TestWorkload (63.14s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [insert violates foreign key constraint] #154715 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [insert violates foreign key constraint] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3.21-rc]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [depends-on relation is dropped during DROP SCHEMA CASCADE] #154221 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [depends-on relation is dropped during DROP SCHEMA CASCADE] [C-test-failure O-robot P-2 T-sql-foundations branch-release-25.3]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [did not received expected DependentObjectsStillExist error] #153823 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [did not received expected DependentObjectsStillExist error] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [column "crdb_internal_column_3_name_placeholder" does not exist] #153126 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [column "crdb_internal_column_3_name_placeholder" does not exist] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3.20-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-55258

[tbg]

on top of 39b6044 I also hit an NPE:

https://mesolite.cluster.engflow.com/invocations/default/a3e14e27-6724-4f97-9a46-047286a50b6b

schema_change_external_test.go:165: Error Trace: pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:165 Error: Received unexpected error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: internal error: while building trigger expression: runtime error: invalid memory address or nil pointer dereference (SQLSTATE XX000) (1) forced error mark | "fatal error when running txn" | github.com/cockroachdb/errors/withstack/*withstack.withStack:: Wraps: (2) Wraps: (3) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt | pkg/workload/schemachange/operation_generator.go:3391 | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn | pkg/workload/schemachange/schemachange.go:523 | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run | pkg/workload/schemachange/schemachange.go:611 | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func5.func7 | pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:154 | golang.org/x/sync/errgroup.(*Group).Go.func1 | external/org_golang_x_sync/errgroup/errgroup.go:93 | runtime.goexit | src/runtime/asm_amd64.s:1700 Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error. Wraps: (5) ERROR: internal error: while building trigger expression: runtime error: invalid memory address or nil pointer dereference (SQLSTATE XX000) Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError Test: TestWorkload

[rafiss]

The original failure was:

{
 "workerId": 0,
 "clientTimestamp": "13:14:14.275878",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER TABLE public.table_w0_141 ALTER PRIMARY KEY USING COLUMNS (col141_w0_147, col141_w0_149, col141_w0_152) USING HASH",
   "potentialExecErr": "2BP01,42701,42P10"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (XX000): executing declarative schema change PostCommitPhase stage 7 of 16 with 3 ValidationType ops (rollback=false) for ALTER TABLE: validation of index crdb_internal_index_4_name_placeholder failed: expected 0 rows, found 4 (SQLSTATE XX000)",

This looks the same as #155038.

[github-actions]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ b136ab6cb76a6cd6eaee03367cffb89624ea9bb9:

     false,
     true,
     true,
     true,
     true,
     true,
     false,
     true
    ]
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w1_3 ALTER PRIMARY KEY USING COLUMNS (col3_w1_8)"
  ]
 }
}
    schema_change_external_test.go:165: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:165
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (XX000): executing declarative schema change PostCommitPhase stage 7 of 15 with 7 ValidationType ops (rollback=false) for ALTER TABLE: validation of index crdb_internal_index_17_name_placeholder failed: expected 0 rows, found 7 (SQLSTATE XX000)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	pkg/workload/schemachange/schemachange.go:682
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func5.func7
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:154
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:93
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1700
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (XX000): executing declarative schema change PostCommitPhase stage 7 of 15 with 7 ValidationType ops (rollback=false) for ALTER TABLE: validation of index crdb_internal_index_17_name_placeholder failed: expected 0 rows, found 7 (SQLSTATE XX000)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 42
Total Schema Statements Succeeded = 42
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 25
Total Transactions Rolled Back = 28
Total Transactions Executed = 53
    schema_change_external_test.go:118: backup, tracing data, and system table dumps in /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/test.outputs/logTestWorkload4002266881
    schema_change_external_test.go:127: running DROP with use_declarative_schema_changer = off
    panic.go:635: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload4002266881
--- FAIL: TestWorkload (32.60s)

Parameters:

• attempt=1
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [insert violates foreign key constraint] #154715 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [insert violates foreign key constraint] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3.21-rc]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [depends-on relation is dropped during DROP SCHEMA CASCADE] #154221 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [depends-on relation is dropped during DROP SCHEMA CASCADE] [C-test-failure O-robot P-2 T-sql-foundations branch-release-25.3]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [did not received expected DependentObjectsStillExist error] #153823 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [did not received expected DependentObjectsStillExist error] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [column "crdb_internal_column_3_name_placeholder" does not exist] #153126 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [column "crdb_internal_column_3_name_placeholder" does not exist] [C-test-failure O-robot P-2 T-sql-foundations branch-release-24.3.20-rc]

This test on roachdash | Improve this report!

[spilchen]

Here is a simple repro:

  CREATE TABLE t (
    a STRING NOT NULL,
    d DECIMAL[],
    INVERTED INDEX (d)
  );

  INSERT INTO t VALUES ('test', ARRAY[1.0, 2.0]);

  ALTER TABLE t ALTER PRIMARY KEY USING COLUMNS (a) USING HASH;

[spilchen]

Git bisect found that this commit, from #154751, introduced the issue: 3e026ce