roachtest: import-cancellation failed #90510
Comments
cockroach-teamcity
added
branch-master
Slightly different to the failure on the 22.2 release branch. |
|
Interestingly, three of the nodes have poor LSM health during the import phase. I wonder if this extended the import phase long enough such that the query phase ran out of time to complete. That might be worth digging into separately.
Node |
|
Did a manual run, which completed in ~10 mins. |
|
I suspect this roachtest's runtime varies significantly, since it uses a prng to determine cancellation, etc. I'm guessing the timed out run involved more or larger import rollbacks. Large import rollbacks can easily cause high r-amp during the import retry, since all of the writes are duplicate writes. All of the ingests into the rolled back keyspace must be ingested into higher levels of the LSM. The import's CheckSSTConflicts calls also become more expensive and no longer hit the fast-paths. For now, I think we should probably edit the test to be deterministic and excercise a fair amount of import rollback. Then we can see what we can do to make this case less painful. I would really like to do cockroachdb/pebble#1170 as soon as we can,, because that would really make a significant difference in this case. |
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Fix cockroachdb#90510. Release note: None.
|
Another thing I'm noticing about the tpc-h workload - the queries are sloooow. For example: SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum((l_extendedprice * (1 - l_discount)) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order
FROM lineitem
WHERE l_shipdate <= ('1998-12-01' - '90 days'::INTERVAL DAY)
GROUP BY l_returnflag, l_linestatus
ORDER BY l_returnflag, l_linestatusHas been running 3200+ seconds. Here's the explain:
There are no stats on that table. I assume with stats we'd be using the index on edit: I misread the explain plan. It is using the right index. However, that particular query has to scan 98% of the table, based on stats I populated on that column manually, and the table is 600M+ rows. For completeness, here's the plan with stats: distribution: full
vectorized: true
• sort
│ estimated row count: 6
│ order: +l_returnflag,+l_linestatus
│
└── • group (hash)
│ estimated row count: 6
│ group by: l_returnflag, l_linestatus
│
└── • render
│
└── • filter
│ estimated row count: 1,183,444,976
│ filter: l_shipdate <= '1998-09-02'
│
└── • scan
estimated row count: 1,199,396,967 (100% of the table; stats collected 14 minutes ago; using stats forecast for 2 hours in the future)
table: lineitem@lineitem_pkey
spans: FULL SCANWith stats, the query finished in a fraction of the time: |
|
roachtest.import-cancellation failed with artifacts on master @ cdf4b18048a9e29019125c64bea75c8d50fa3774: Parameters: Same failure on other branches
|
|
roachtest.import-cancellation failed with artifacts on master @ 11e299bb1bec2f5666658393f14c12ebe11cc4a4: Parameters: Same failure on other branches
|
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Additionally, reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Fix cockroachdb#90510. Release note: None.
|
Last night's run was also a timeout. The breakdown of the time spent is roughly as follows:
The failure prior to that was due to some context cancellation - still digging into that one. |
celiala
added
GA-blocker
blocks-22.2.0-rc.1
and removed
release-blocker
|
Marking as GA blocker, as per thread. |
|
roachtest.import-cancellation failed with artifacts on master @ c5decaeb69155399647909163451f3d8ca7858b6: Parameters: Same failure on other branches
|
|
I'm going to test another patch to this test that forces table stats collection, waits for the jobs to complete, and then see what difference that makes to the workload runtime. I suspect the stats collection job will be useful, as it will need to scan the whole keyspace. |
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Force the collection of table statistics as soon as the table has finished importing. Fix cockroachdb#90510. Release note: None.
|
roachtest.import-cancellation failed with artifacts on master @ 1b1c8da55be48c174b7b370b305f42622546209f: Parameters: Same failure on other branches
|
90547: roachtest: import-cancellation: use a fixed seed; smaller import r=jbowens a=nicktrav Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Additionally, reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Fix #90510. Release note: None. 90720: Revert fmtsafe.go output r=knz a=smg260 TestLint/TestRoachVet is a unit test which runs roachvet and greps various messages from its output - the reverted message being one of them. Release note: none. Epic: none 90736: sqlinstance: don't pretend to release the instance ID on shutdown r=andreimatei a=andreimatei Before this patch, the code looked like tenant SQL servers release their instance IDs on shutdown, but in fact they didn't. Instead of releasing on server shutdown, releasing the instance ID was done in a callback that was running under very specific circumstances - i.e. when the session heartbeat loop found the session record to have been deleted. Additionally, that callback also ran unreliably - it short-circuited if the session is not expired according to the local clock. This patch does away with releasing the instance ID, on two arguments: 1. The instance ID was rarely and inconsistently released to begin with, as explained above. 2. The instance ID release code is buggy - it blindly deletes the instance ID, even if another server had stolen it. In fact, given when we were releasing, I think we were particularly likely to release IDs that had been stolen. This is very dangerous. Better to never do it than run this risk. I've considered properly releasing the instance ID on shutdown, but I had trouble working out exactly when and where this can be done: you can't do it after stopper quiescing, and you don't want to do it too soon before that while the instance ID is still used by the server. I think properly releasing the ID will have to be coupled with a way of rejecting further uses of the ID. In the meantime I've left a TODO. Fixes #90576 Release note: None Co-authored-by: Nick Travers <travers@cockroachlabs.com> Co-authored-by: Miral Gadani <miral@cockroachlabs.com> Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Force the collection of table statistics as soon as the table has finished importing. Reduce the number of time each TPC-H query is repeated, from three to two. Fix #90510. Release note: None.
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Force the collection of table statistics as soon as the table has finished importing. Reduce the number of time each TPC-H query is repeated, from three to two. Fix cockroachdb#90510. Release note: None.
Currently, due to the randomness present in the test, there may be significant variance in test runs. In some situations, the nature of the random import cancellations may result in the test timing out. This makes the test flaky. Use a fixed seed for the test's PRNG to make the test more deterministic. Reduce the size of the overall import by lowering the number of files used to generate the `lineitem` table. This should also help stabilize the runtime. Force the collection of table statistics as soon as the table has finished importing. Reduce the number of time each TPC-H query is repeated, from three to two. Fix cockroachdb#90510. Release note: None.
nicktrav
added
C-bug
roachtest.import-cancellation failed with artifacts on master @ a3b578466bcdcc5d2bbea628d33c337d87b9f4b8:
Parameters:
ROACHTEST_cloud=gce,ROACHTEST_cpu=32,ROACHTEST_encrypted=false,ROACHTEST_fs=ext4,ROACHTEST_localSSD=true,ROACHTEST_ssd=0Help
See: roachtest README
See: How To Investigate (internal)
Same failure on other branches
This test on roachdash | Improve this report!
Jira issue: CRDB-20809
Epic CRDB-16237
The text was updated successfully, but these errors were encountered: