roachtest: import-cancellation failed #90434

cockroach-teamcity · 2022-10-21T12:12:55Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ 8579361117ff18a4255855a43114ddeb3e3b80c9:

		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | pplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  |  1108.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1108.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1109.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1109.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  |  1110.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1110.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:163,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/storage _{This test on roachdash | Improve this report!

Jira issue: CRDB-20767
Epic CRDB-16237}

The text was updated successfully, but these errors were encountered:

jbowens · 2022-10-21T13:42:59Z

Looks like the GC threshold is too aggressive for these large tpch queries. We can bump the threshold after the import succeeds.

Error: pq: batch timestamp 1666353617.904578055,0 must be after replica GC threshold 1666353685.928197966,0
run_115351.234765469_n1_workload_run_querybench: 12:12:24 cluster.go:1999: > result: ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s returned: COMMAND_PROBLEM: exit status 1
(1) ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s returned
  | stderr:
  | I221021 11:53:53.462830 1 workload/cli/run.go:427  [-] 1  creating load generator...
  | I221021 11:53:53.634337 1 workload/cli/run.go:458  [-] 2  creating load generator... done (took 171.545659ms)
  | Error: pq: batch timestamp 1666353617.904578055,0 must be after replica GC threshold 1666353685.928197966,0

cockroach-teamcity · 2022-10-22T12:10:53Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ fc582bd2d43c1dc141077f2fae6da0ca9c23e449:

		  | Error: pq: batch timestamp 1666439623.970102632,0 must be after replica GC threshold 1666439688.063994730,0
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  |  1399.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1399.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  |  1400.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1400.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		  |  1401.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  1: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM lineitem WHERE l_shipdate <= DATE '1998-12-01' - INTERVAL '90' DAY GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus
		  |  1401.0s        0            0.0            0.0      0.0      0.0      0.0      0.0  2: SELECT s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment FROM part, supplier, partsupp, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND p_size = 15 AND p_type LIKE '%BRASS' AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE' AND ps_supplycost = ( SELECT min(ps_supplycost) FROM partsupp, supplier, nation, region WHERE p_partkey = ps_partkey AND s_suppkey = ps_suppkey AND s_nationkey = n_nationkey AND n_regionkey = r_regionkey AND r_name = 'EUROPE') ORDER BY s_acctbal DESC, n_name, s_name, p_partkey LIMIT 100
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:163,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

nicktrav · 2022-10-22T14:45:55Z

Failure mode looks the same:

  | Error: pq: batch timestamp 1666439623.970102632,0 must be after replica GC threshold 1666439688.063994730,0

Should be fixed with #90487.

cockroach-teamcity · 2022-10-23T14:22:41Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ c0e9c25661d53e893210a7bfe79257f1c9468f52:

		  | , lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderDATE >= DATE '1993-10-01' AND o_orderDATE < DATE '1993-10-01' + INTERVAL '3' MONTH AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC LIMIT 20
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 11: SELECT ps_partkey, sum(ps_supplycost * ps_availqty::float) AS value FROM partsupp, supplier, nation WHERE ps_suppkey = s_suppkey AND s_nationkey = n_nationkey AND n_name = 'GERMANY' GROUP BY ps_partkey HAVING sum(ps_supplycost * ps_availqty::float) > ( SELECT sum(ps_supplycost * ps_availqty::float) * 0.0001 FROM partsupp, supplier, nation WHERE ps_suppkey = s_suppkey AND s_nationkey = n_nationkey AND n_name = 'GERMANY') ORDER BY value DESC
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 12: SELECT l_shipmode, sum(CASE WHEN o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END) AS high_line_count, sum(CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= DATE '1994-01-01' AND l_receiptdate < DATE '1994-01-01' + INTERVAL '1' YEAR GROUP BY l_shipmode ORDER BY l_shipmode
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 13: SELECT c_count, count(*) AS custdist FROM ( SELECT c_custkey, count(o_orderkey) AS c_count FROM customer LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' GROUP BY c_custkey) AS c_orders GROUP BY c_count ORDER BY custdist DESC, c_count DESC
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 14: SELECT 100.00 * sum(CASE WHEN p_type LIKE 'PROMO%' THEN l_extendedprice * (1 - l_discount) ELSE 0 END) / sum(l_extendedprice * (1 - l_discount)) AS promo_revenue FROM lineitem, part WHERE l_partkey = p_partkey AND l_shipdate >= DATE '1995-09-01' AND l_shipdate < DATE '1995-09-01' + INTERVAL '1' MONTH
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 15: CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, sum(l_extendedprice * (1 - l_discount)) FROM lineitem WHERE l_shipdate >= DATE '1996-01-01' AND l_shipdate < DATE '1996-01-01' + INTERVAL '3' MONTH GROUP BY l_suppkey; SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = ( SELECT max(total_revenue) FROM revenue0) ORDER BY s_suppkey; DROP VIEW revenue0
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 16: SELECT p_brand, p_type, p_size, count(distinct ps_suppkey) AS supplier_cnt FROM partsupp, part WHERE p_partkey = ps_partkey AND p_brand <> 'Brand#45' AND p_type NOT LIKE 'MEDIUM POLISHED%' AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9) AND ps_suppkey NOT IN ( SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%') GROUP BY p_brand, p_type, p_size ORDER BY supplier_cnt DESC, p_brand, p_type, p_size
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 17: SELECT sum(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 0.2 * avg(l_quantity) FROM lineitem WHERE l_partkey = p_partkey)
		  |  8770.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 18: SELECT c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem GROUP BY l_orderkey HAVING sum(l_quantity) > 300) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice ORDER BY o_totalprice DESC, o_orderdate LIMIT 100
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:174,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:174
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1061,test_runner.go:960: test timed out (4h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import-cancellation failed #90510 roachtest: import-cancellation failed [C-test-failure O-roachtest O-robot T-storage branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-10-24T14:25:54Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ 794fcfa793016e98a6ebadaaa29444e4661592ba:

		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |        0.0      0.0      0.0      0.0      0.0 13: SELECT c_count, count(*) AS custdist FROM ( SELECT c_custkey, count(o_orderkey) AS c_count FROM customer LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' GROUP BY c_custkey) AS c_orders GROUP BY c_count ORDER BY custdist DESC, c_count DESC
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 14: SELECT 100.00 * sum(CASE WHEN p_type LIKE 'PROMO%' THEN l_extendedprice * (1 - l_discount) ELSE 0 END) / sum(l_extendedprice * (1 - l_discount)) AS promo_revenue FROM lineitem, part WHERE l_partkey = p_partkey AND l_shipdate >= DATE '1995-09-01' AND l_shipdate < DATE '1995-09-01' + INTERVAL '1' MONTH
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 15: CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, sum(l_extendedprice * (1 - l_discount)) FROM lineitem WHERE l_shipdate >= DATE '1996-01-01' AND l_shipdate < DATE '1996-01-01' + INTERVAL '3' MONTH GROUP BY l_suppkey; SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = ( SELECT max(total_revenue) FROM revenue0) ORDER BY s_suppkey; DROP VIEW revenue0
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 16: SELECT p_brand, p_type, p_size, count(distinct ps_suppkey) AS supplier_cnt FROM partsupp, part WHERE p_partkey = ps_partkey AND p_brand <> 'Brand#45' AND p_type NOT LIKE 'MEDIUM POLISHED%' AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9) AND ps_suppkey NOT IN ( SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%') GROUP BY p_brand, p_type, p_size ORDER BY supplier_cnt DESC, p_brand, p_type, p_size
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 17: SELECT sum(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 0.2 * avg(l_quantity) FROM lineitem WHERE l_partkey = p_partkey)
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 18: SELECT c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem GROUP BY l_orderkey HAVING sum(l_quantity) > 300) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice ORDER BY o_totalprice DESC, o_orderdate LIMIT 100
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 19: SELECT sum(l_extendedprice* (1 - l_discount)) AS revenue FROM lineitem, part WHERE ( p_partkey = l_partkey AND p_brand = 'Brand#12' AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') AND l_quantity >= 1 AND l_quantity <= 1 + 10 AND p_size BETWEEN 1 AND 5 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') AND l_quantity >= 10 AND l_quantity <= 10 + 10 AND p_size BETWEEN 1 AND 10 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON')
		  |  9471.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 20: SELECT s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%') AND ps_availqty > ( SELECT 0.5 * sum(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1994-01-01' + INTERVAL '1' YEAR)) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:174,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:174
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1061,test_runner.go:960: test timed out (4h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import-cancellation failed #90510 roachtest: import-cancellation failed [C-test-failure O-roachtest O-robot T-storage branch-master release-blocker]

_{This test on roachdash | Improve this report!}

nicktrav · 2022-10-24T18:19:51Z

For the two previous failures, both appear to be test timeouts:

Run n: ~1m spin-up + 10:26-11:47 (1h21m) import + 11:47-14:25 (2h38m).
Run n-1: <1m spin-up + 10:22-11:55 (1h33m) import + 11:55-14:22 (2h27m).

jbowens · 2022-10-24T19:21:29Z

I tried running the roachtest with range tombstones disabled. The import took 25m. The workload just started.

It looks like we're seeing a 3-4x slowdown of the IMPORT phase when using range tombstones. Some of this is expected because we're no longer IMPORTing into a clean empty span: we're importing over a span full of garbage. This increases the cost of CheckSSTConflicts, but it also greatly increases the work the storage engine must do. All the ingested files have significant overlap, so they must be ingested into higher levels. This necessitates more compaction work to reshape the LSM, and is why we see elevated read amp.

Unfortunately, I don't think we can do anything about this for 22.2. We should consider what we can do here in 23.1, but it is just a legitimately hard problem.

nicktrav · 2022-10-24T20:01:22Z

I forgot to post this last week. I took a few CPU profiles during the import phase. Nothing to unexpected here re: CheckSSTConflicts, but in case it helps shed some more light on where we can optimize:

jbowens · 2022-10-24T22:00:38Z

Followup on this run #90434 (comment):

The test failed with a timeout even with range tombstones disabled. Like @nicktrav noticed, it looks like the tpch workload has its own variance (maybe due to whether or not it has stats computed in time?).

jbowens · 2022-10-24T22:04:21Z

@nicktrav that CPU profile demonstrates the pathological behavior of range key seeks as a result of seek semantics. I think cockroachdb/pebble#1829 will make a large difference there. I'm hoping to tackle that first as a part of the range key cleanup work this cycle.

cockroach-teamcity · 2022-10-25T14:23:21Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ 6df4e8666ccb902b333195bdded59d69307b6050:

		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 10: SELECT c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) AS revenue, c_acctbal, n_name, c_address, c_phone, c_comment FROM customer, orders, lineitem, nation WHERE c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderDATE >= DATE '1993-10-01' AND o_orderDATE < DATE '1993-10-01' + INTERVAL '3' MONTH AND l_returnflag = 'R' AND c_nationkey = n_nationkey GROUP BY c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment ORDER BY revenue DESC LIMIT 20
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 11: SELECT ps_partkey, sum(ps_supplycost * ps_availqty::float) AS value FROM partsupp, supplier, nation WHERE ps_suppkey = s_suppkey AND s_nationkey = n_nationkey AND n_name = 'GERMANY' GROUP BY ps_partkey HAVING sum(ps_supplycost * ps_availqty::float) > ( SELECT sum(ps_supplycost * ps_availqty::float) * 0.0001 FROM partsupp, supplier, nation WHERE ps_suppkey = s_suppkey AND s_nationkey = n_nationkey AND n_name = 'GERMANY') ORDER BY value DESC
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 12: SELECT l_shipmode, sum(CASE WHEN o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH' THEN 1 ELSE 0 END) AS high_line_count, sum(CASE WHEN o_orderpriority <> '1-URGENT' AND o_orderpriority <> '2-HIGH' THEN 1 ELSE 0 END) AS low_line_count FROM orders, lineitem WHERE o_orderkey = l_orderkey AND l_shipmode IN ('MAIL', 'SHIP') AND l_commitdate < l_receiptdate AND l_shipdate < l_commitdate AND l_receiptdate >= DATE '1994-01-01' AND l_receiptdate < DATE '1994-01-01' + INTERVAL '1' YEAR GROUP BY l_shipmode ORDER BY l_shipmode
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 13: SELECT c_count, count(*) AS custdist FROM ( SELECT c_custkey, count(o_orderkey) AS c_count FROM customer LEFT OUTER JOIN orders ON c_custkey = o_custkey AND o_comment NOT LIKE '%special%requests%' GROUP BY c_custkey) AS c_orders GROUP BY c_count ORDER BY custdist DESC, c_count DESC
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 14: SELECT 100.00 * sum(CASE WHEN p_type LIKE 'PROMO%' THEN l_extendedprice * (1 - l_discount) ELSE 0 END) / sum(l_extendedprice * (1 - l_discount)) AS promo_revenue FROM lineitem, part WHERE l_partkey = p_partkey AND l_shipdate >= DATE '1995-09-01' AND l_shipdate < DATE '1995-09-01' + INTERVAL '1' MONTH
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 15: CREATE VIEW revenue0 (supplier_no, total_revenue) AS SELECT l_suppkey, sum(l_extendedprice * (1 - l_discount)) FROM lineitem WHERE l_shipdate >= DATE '1996-01-01' AND l_shipdate < DATE '1996-01-01' + INTERVAL '3' MONTH GROUP BY l_suppkey; SELECT s_suppkey, s_name, s_address, s_phone, total_revenue FROM supplier, revenue0 WHERE s_suppkey = supplier_no AND total_revenue = ( SELECT max(total_revenue) FROM revenue0) ORDER BY s_suppkey; DROP VIEW revenue0
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 16: SELECT p_brand, p_type, p_size, count(distinct ps_suppkey) AS supplier_cnt FROM partsupp, part WHERE p_partkey = ps_partkey AND p_brand <> 'Brand#45' AND p_type NOT LIKE 'MEDIUM POLISHED%' AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9) AND ps_suppkey NOT IN ( SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%') GROUP BY p_brand, p_type, p_size ORDER BY supplier_cnt DESC, p_brand, p_type, p_size
		  |  9086.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 17: SELECT sum(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 0.2 * avg(l_quantity) FROM lineitem WHERE l_partkey = p_partkey)
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:174,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:174
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1061,test_runner.go:960: test timed out (4h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import-cancellation failed #90510 roachtest: import-cancellation failed [C-test-failure GA-blocker O-roachtest O-robot T-storage blocks-22.2.0-rc.1 branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-10-26T14:18:34Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ 890b681dd5bb2c5ebabb06d89aed131af0b50fe8:

		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 16: SELECT p_brand, p_type, p_size, count(distinct ps_suppkey) AS supplier_cnt FROM partsupp, part WHERE p_partkey = ps_partkey AND p_brand <> 'Brand#45' AND p_type NOT LIKE 'MEDIUM POLISHED%' AND p_size IN (49, 14, 23, 45, 19, 3, 36, 9) AND ps_suppkey NOT IN ( SELECT s_suppkey FROM supplier WHERE s_comment LIKE '%Customer%Complaints%') GROUP BY p_brand, p_type, p_size ORDER BY supplier_cnt DESC, p_brand, p_type, p_size
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 17: SELECT sum(l_extendedprice) / 7.0 AS avg_yearly FROM lineitem, part WHERE p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container = 'MED BOX' AND l_quantity < ( SELECT 0.2 * avg(l_quantity) FROM lineitem WHERE l_partkey = p_partkey)
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 18: SELECT c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) FROM customer, orders, lineitem WHERE o_orderkey IN ( SELECT l_orderkey FROM lineitem GROUP BY l_orderkey HAVING sum(l_quantity) > 300) AND c_custkey = o_custkey AND o_orderkey = l_orderkey GROUP BY c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice ORDER BY o_totalprice DESC, o_orderdate LIMIT 100
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 19: SELECT sum(l_extendedprice* (1 - l_discount)) AS revenue FROM lineitem, part WHERE ( p_partkey = l_partkey AND p_brand = 'Brand#12' AND p_container IN ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') AND l_quantity >= 1 AND l_quantity <= 1 + 10 AND p_size BETWEEN 1 AND 5 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#23' AND p_container IN ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') AND l_quantity >= 10 AND l_quantity <= 10 + 10 AND p_size BETWEEN 1 AND 10 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON') OR ( p_partkey = l_partkey AND p_brand = 'Brand#34' AND p_container IN ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') AND l_quantity >= 20 AND l_quantity <= 20 + 10 AND p_size BETWEEN 1 AND 15 AND l_shipmode IN ('AIR', 'AIR REG') AND l_shipinstruct = 'DELIVER IN PERSON')
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 20: SELECT s_name, s_address FROM supplier, nation WHERE s_suppkey IN ( SELECT ps_suppkey FROM partsupp WHERE ps_partkey IN ( SELECT p_partkey FROM part WHERE p_name LIKE 'forest%') AND ps_availqty > ( SELECT 0.5 * sum(l_quantity) FROM lineitem WHERE l_partkey = ps_partkey AND l_suppkey = ps_suppkey AND l_shipdate >= DATE '1994-01-01' AND l_shipdate < DATE '1994-01-01' + INTERVAL '1' YEAR)) AND s_nationkey = n_nationkey AND n_name = 'CANADA' ORDER BY s_name
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 21: SELECT s_name, count(*) AS numwait FROM supplier, lineitem l1, orders, nation WHERE s_suppkey = l1.l_suppkey AND o_orderkey = l1.l_orderkey AND o_orderstatus = 'F' AND l1.l_receiptDATE > l1.l_commitdate AND EXISTS ( SELECT * FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey) AND NOT EXISTS ( SELECT * FROM lineitem l3 WHERE l3.l_orderkey = l1.l_orderkey AND l3.l_suppkey <> l1.l_suppkey AND l3.l_receiptDATE > l3.l_commitdate) AND s_nationkey = n_nationkey AND n_name = 'SAUDI ARABIA' GROUP BY s_name ORDER BY numwait DESC, s_name LIMIT 100
		  |  9311.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 22: SELECT cntrycode, count(*) AS numcust, sum(c_acctbal) AS totacctbal FROM ( SELECT substring(c_phone FROM 1 FOR 2) AS cntrycode, c_acctbal FROM customer WHERE substring(c_phone FROM 1 FOR 2) in ('13', '31', '23', '29', '30', '18', '17') AND c_acctbal > ( SELECT avg(c_acctbal) FROM customer WHERE c_acctbal > 0.00 AND substring(c_phone FROM 1 FOR 2) in ('13', '31', '23', '29', '30', '18', '17')) AND NOT EXISTS ( SELECT * FROM orders WHERE o_custkey = c_custkey)) AS custsale GROUP BY cntrycode ORDER BY cntrycode
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run querybench --db=csv --concurrency=1 --query-file=tpch --num-runs=3 --max-ops=66 {pgurl:1-6} --histograms=perf/stats.json --histograms-max-latency=8m20s
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

	monitor.go:127,import_cancellation.go:174,import_cancellation.go:38,test_runner.go:930: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:174
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	main/pkg/cmd/roachtest/monitor.go:80
		  | runtime.doInit
		  | 	GOROOT/src/runtime/proc.go:6340
		  | runtime.main
		  | 	GOROOT/src/runtime/proc.go:233
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

	test_runner.go:1061,test_runner.go:960: test timed out (4h0m0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: import-cancellation failed #90510 roachtest: import-cancellation failed [C-test-failure GA-blocker O-roachtest O-robot T-storage blocks-22.2.0-rc.1 branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-10-27T10:46:47Z

roachtest.import-cancellation failed with artifacts on release-22.2 @ 4816df3a9d76d179ed135a2b1efb53babb5611a0:

test artifacts and logs in: /artifacts/import-cancellation/run_1
	import_cancellation.go:295,import_cancellation.go:137,monitor.go:105,errgroup.go:74: Job 808703287385423875 failed.
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*importCancellationTest).runImportSequence
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:295
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runImportCancellation.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import_cancellation.go:137
		  | main.(*monitorImpl).Go.func1
		  | 	main/pkg/cmd/roachtest/monitor.go:105
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) Job 808703287385423875 failed.
		Error types: (1) *withstack.withStack (2) *errutil.leafError

	import_cancellation.go:282,import_cancellation.go:137,monitor.go:105,errgroup.go:74: pq: query execution canceled

	import_cancellation.go:150,import_cancellation.go:38,test_runner.go:930: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

nicktrav · 2022-10-27T18:14:02Z

This should be fixed via #90741. Closing.

cockroach-teamcity added this to the 22.2 milestone Oct 21, 2022

blathers-crl bot added the T-storage Storage Team label Oct 21, 2022

jbowens removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 21, 2022

cockroach-teamcity mentioned this issue Oct 21, 2022

roachtest: import-cancellation failed #90445

Closed

jbowens mentioned this issue Oct 21, 2022

release-22.2: cmd/roachtest: fix GC TTL in import-cancellation test #90487

Merged

cockroach-teamcity mentioned this issue Oct 22, 2022

roachtest: import-cancellation failed #90510

Closed

nicktrav self-assigned this Oct 24, 2022

nicktrav closed this as completed Oct 27, 2022

nicktrav added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-storage Relating to our storage engine (Pebble) on-disk storage. labels Dec 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: import-cancellation failed #90434

roachtest: import-cancellation failed #90434

cockroach-teamcity commented Oct 21, 2022 •

edited by exalate-issue-sync bot

Loading

jbowens commented Oct 21, 2022

cockroach-teamcity commented Oct 22, 2022

nicktrav commented Oct 22, 2022

cockroach-teamcity commented Oct 23, 2022

cockroach-teamcity commented Oct 24, 2022

nicktrav commented Oct 24, 2022

jbowens commented Oct 24, 2022

nicktrav commented Oct 24, 2022

jbowens commented Oct 24, 2022

jbowens commented Oct 24, 2022

cockroach-teamcity commented Oct 25, 2022

cockroach-teamcity commented Oct 26, 2022

cockroach-teamcity commented Oct 27, 2022

nicktrav commented Oct 27, 2022

roachtest: import-cancellation failed #90434

roachtest: import-cancellation failed #90434

Comments

cockroach-teamcity commented Oct 21, 2022 • edited by exalate-issue-sync bot Loading

jbowens commented Oct 21, 2022

cockroach-teamcity commented Oct 22, 2022

nicktrav commented Oct 22, 2022

cockroach-teamcity commented Oct 23, 2022

cockroach-teamcity commented Oct 24, 2022

nicktrav commented Oct 24, 2022

jbowens commented Oct 24, 2022

nicktrav commented Oct 24, 2022

jbowens commented Oct 24, 2022

jbowens commented Oct 24, 2022

cockroach-teamcity commented Oct 25, 2022

cockroach-teamcity commented Oct 26, 2022

cockroach-teamcity commented Oct 27, 2022

nicktrav commented Oct 27, 2022

cockroach-teamcity commented Oct 21, 2022 •

edited by exalate-issue-sync bot

Loading