Skip to content

fix(ci): break shard ties on bucket count so a thin durations file cannot collapse (#14802) - #14803

Merged
mrveiss merged 1 commit into
Dev_new_guifrom
issue-14802-shard-tiebreak
Aug 22, 2026
Merged

mrveiss merged 1 commit into
Dev_new_guifrom
issue-14802-shard-tiebreak

Conversation

@mrveiss

@mrveiss mrveiss commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Closes #14802. Blocks-fix for #14648.

Thinking Path

build_bucket_table assigns buckets to shards by greedy longest-processing-time:

target = min(range(splits), key=lambda s: (shard_load[s], s))
table[bucket] = target
shard_load[target] += bucket_weight[bucket]

A zero-weight bucket adds nothing to shard_load. So once a shard holds the
minimum load it keeps winning the tie, and every weightless bucket after it lands on
that same shard. The shards past it receive nothing.

Reproduced against a standalone reimplementation of the published algorithm, not by
running repo code:

durations input shards used, before after
empty / missing 1 of 12 12 of 12
one module 2 of 12 12 of 12
five modules 6 of 12 12 of 12
realistic, ~1282 modules 12 of 12 12 of 12

The important part is that it is structural, not transient. Assignment is by
hash, so a shard that receives no bucket cannot be reached by any file added later
either. Under the one-module case, 5000 hypothetical new modules hashed into shards
[0, 1] only.

Why it was invisible

The committed .test_durations describes ~1282 modules, so all twelve shards are
used and the spread is fine. The collapse needs a missing, empty, truncated or very
thin durations file — and those are committed by hand from a workflow artifact, with
no non-degeneracy check before commit.

The existing test looked like it covered this:

assert set(table) <= set(range(SPLITS))

<= is satisfied when one shard of twelve is used. The degenerate case read as
covered while being untested.

Why it matters now

#14648 is what activates this algorithm in CI. Its SLM step tolerates pytest's
exit 5 (no tests collected) and returns 0 — written for the legitimate "fewer
modules than shards" case. Combined with a collapse, eleven of twelve SLM shards
would each exit 0 with a ::notice::, one shard would silently run the whole suite,
and the aggregator would report that every shard passed.

That is worse than the exit-5 redness this repo has already been bitten by: it turns
the failure green instead of red.

It is also a regression against the old behaviour — pytest-split degrades to
equal-count contiguous chunks across all groups when durations are missing.

What Changed

Tie-break on a per-shard bucket count as well as load, incremented unconditionally,
so weightless buckets round-robin instead of piling up.

Verification

Each new test run against both algorithms:

no-durations   OLD: FAIL (1/12)    NEW: PASS (12/12)
one-module     OLD: FAIL (2/12)    NEW: PASS (12/12)
five-modules   OLD: FAIL (6/12)    NEW: PASS (12/12)
future modules OLD: FAIL (reached [0, 1])   NEW: PASS (reached all 12)
realistic      OLD: spread 1.001x  NEW: spread 1.001x   (unchanged)

Every one fails against the old tie-break, so none of them is decorative, and the
realistic distribution is untouched — this fixes the degenerate case without
perturbing the balanced one.

Model Used

Claude Opus 5

…nnot collapse (#14802)

A zero-weight bucket adds nothing to shard_load, so the load-only tie-break handed
the same lowest-index shard every subsequent tie and every weightless bucket piled
onto it. With an empty durations file all 512 buckets landed on shard 0; with one
module, on shards 0 and 1.

It is structural rather than transient: the mapping is by hash, so the shards that
receive nothing stay unreachable by any file that could later be added, not merely
empty today. 5000 hypothetical new modules reached only shards 0 and 1.

Today's committed durations describes ~1282 modules, so every shard is used and the
spread is unchanged at 1.001x in a realistic distribution — which is exactly why
this was invisible.

It matters now because #14648 activates this algorithm in CI, and the SLM step
tolerates pytest's exit 5, so collapsed-empty shards would each report success and
the aggregator would say every shard passed. Green, not red.

The existing test asserted `set(table) <= set(range(SPLITS))`, which holds when one
shard of twelve is used, so the case read as covered while being untested. It now
asserts equality, with explicit no-durations, one-module and five-module cases and
a check that future modules can still reach every shard. All four fail against the
old tie-break and pass with this one.
@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

@mrveiss
mrveiss merged commit 597ccc2 into Dev_new_gui Aug 22, 2026
56 checks passed
@mrveiss
mrveiss deleted the issue-14802-shard-tiebreak branch August 22, 2026 13:57
mrveiss added a commit that referenced this pull request Aug 22, 2026
… (#14648)

Both python-suite targets now shard by the stable hash-bucket assignment rather
than pytest-split, so a module keeps its shard as the collected set changes.

Merged after #14803, which fixed a collapse in that algorithm: a zero-weight bucket
did not move shard_load, so every weightless bucket piled onto the lowest-index
shard and a thin durations file left most shards permanently unreachable. Landing
this first would have activated that in CI, where the SLM exit-5 tolerance would
have reported the collapsed shards as success.

Also fixes a sys.modules leak this reshuffle surfaced. The autouse fixture filtered
plugin_sdk aliases through a per-test  snapshot, but the loader installs
them process-wide, so whichever test loads a plugin first in the shard owns them and
every later test sees them as pre-existing. They are now removed unconditionally,
which is safe because the loader re-creates them on demand.

Not baselined — the leak is fixed, not silenced.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant