roachtest: rebalance/by-load/replicas/mixed-version failed [hits hyperloglog crash #137386]

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(mixedversion.go:732).Run: mixed-version test failure while running step 26 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=32.0 tolerance=20.0% (±6.4) bounds=[25.6, 38.3]
	below  = [s3: 0 (-100.0%), s4: 1 (-95.3%), s5: 7 (-77.2%)]
	within = []
	above  = [s1: 51 (+60.5%), s2: 84 (+163.8%), s6: 47 (+48.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: rebalance/by-load/replicas/mixed-version failed #133004 roachtest: rebalance/by-load/replicas/mixed-version failed [C-test-failure O-roachtest O-robot T-kv branch-release-24.3 release-blocker]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-43443

[arulajmani]

[mixed-version-test] 2024/10/21 12:31:01 mixedversion.go:729: mixed-version test:
Seed:               -3544064184599451786
Upgrades:           v23.1.28 → v23.2.12 → v24.1.5 → v24.2.3 → master
Deployment mode:    shared-process
Mutators:           preserve_downgrade_option_randomizer
Plan:

Given this is shared-process multi-tenancy, I'll remove the release-blocker label.

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 9c1d89e3adb8c6532459cc3e616288db06f966d9:

(mixedversion.go:732).Run: mixed-version test failure while running step 12 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=56.4 tolerance=20.0% (±11.3) bounds=[45.1, 67.6]
	below  = [s4: 0 (-100.0%), s5: 21 (-61.7%)]
	within = [s6: 56 (-0.5%)]
	above  = [s1: 72 (+28.4%), s2: 90 (+59.9%), s3: 97 (+73.9%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

• roachtest: rebalance/by-load/replicas/mixed-version failed #133004 roachtest: rebalance/by-load/replicas/mixed-version failed [C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 82b1fda15c4616713b278c447d24b0ab5416e511:

(mixedversion.go:732).Run: mixed-version test failure while running step 45 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=7.5 tolerance=20.0% (±1.5) bounds=[6.0, 9.0]
	below  = [s1: 0 (-100.0%), s2: 5 (-29.6%), s3: 4 (-42.3%), s4: 3 (-54.9%), s5: 5 (-27.0%)]
	within = []
	above  = [s6: 26 (+253.8%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

• roachtest: rebalance/by-load/replicas/mixed-version failed #133004 roachtest: rebalance/by-load/replicas/mixed-version failed [C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3]

This test on roachdash | Improve this report!

[miraradeva]

Adding C-bug label to move from the triage queue.

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 7495c0390f93478bc0f961285f41cd4ec85a6a76:

(mixedversion.go:732).Run: mixed-version test failure while running step 27 (restart node 1 with binary version v24.1.6): waiting for shared-process tenant on n1: pq: internal error while retrieving user account memberships: operation "get-user-session" timed out after 10s (given timeout 10s): internal error while retrieving user account: get auth info error: interrupted during singleflight load-value:authinfo-roachprod-2-2: context deadline exceeded
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

• roachtest: rebalance/by-load/replicas/mixed-version failed #133004 roachtest: rebalance/by-load/replicas/mixed-version failed [C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.3]

This test on roachdash | Improve this report!