Repository navigation
fix(ci): break shard ties on bucket count so a thin durations file cannot collapse (#14802) - #14803
Merged
Merged
Conversation
…nnot collapse (#14802) A zero-weight bucket adds nothing to shard_load, so the load-only tie-break handed the same lowest-index shard every subsequent tie and every weightless bucket piled onto it. With an empty durations file all 512 buckets landed on shard 0; with one module, on shards 0 and 1. It is structural rather than transient: the mapping is by hash, so the shards that receive nothing stay unreachable by any file that could later be added, not merely empty today. 5000 hypothetical new modules reached only shards 0 and 1. Today's committed durations describes ~1282 modules, so every shard is used and the spread is unchanged at 1.001x in a realistic distribution — which is exactly why this was invisible. It matters now because #14648 activates this algorithm in CI, and the SLM step tolerates pytest's exit 5, so collapsed-empty shards would each report success and the aggregator would say every shard passed. Green, not red. The existing test asserted `set(table) <= set(range(SPLITS))`, which holds when one shard of twelve is used, so the case read as covered while being untested. It now asserts equality, with explicit no-durations, one-module and five-module cases and a check that future modules can still reach every shard. All four fail against the old tie-break and pass with this one.
Contributor
✅ SSOT Configuration Compliance: Passing🎉 No hardcoded values detected that have SSOT config equivalents! |
Closed
4 tasks
mrveiss
added a commit
that referenced
this pull request
Aug 22, 2026
… (#14648) Both python-suite targets now shard by the stable hash-bucket assignment rather than pytest-split, so a module keeps its shard as the collected set changes. Merged after #14803, which fixed a collapse in that algorithm: a zero-weight bucket did not move shard_load, so every weightless bucket piled onto the lowest-index shard and a thin durations file left most shards permanently unreachable. Landing this first would have activated that in CI, where the SLM exit-5 tolerance would have reported the collapsed shards as success. Also fixes a sys.modules leak this reshuffle surfaced. The autouse fixture filtered plugin_sdk aliases through a per-test snapshot, but the loader installs them process-wide, so whichever test loads a plugin first in the shard owns them and every later test sees them as pre-existing. They are now removed unconditionally, which is safe because the loader re-creates them on demand. Not baselined — the leak is fixed, not silenced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14802. Blocks-fix for #14648.
Thinking Path
build_bucket_tableassigns buckets to shards by greedy longest-processing-time:A zero-weight bucket adds nothing to
shard_load. So once a shard holds theminimum load it keeps winning the tie, and every weightless bucket after it lands on
that same shard. The shards past it receive nothing.
Reproduced against a standalone reimplementation of the published algorithm, not by
running repo code:
The important part is that it is structural, not transient. Assignment is by
hash, so a shard that receives no bucket cannot be reached by any file added later
either. Under the one-module case, 5000 hypothetical new modules hashed into shards
[0, 1]only.Why it was invisible
The committed
.test_durationsdescribes ~1282 modules, so all twelve shards areused and the spread is fine. The collapse needs a missing, empty, truncated or very
thin durations file — and those are committed by hand from a workflow artifact, with
no non-degeneracy check before commit.
The existing test looked like it covered this:
<=is satisfied when one shard of twelve is used. The degenerate case read ascovered while being untested.
Why it matters now
#14648 is what activates this algorithm in CI. Its SLM step tolerates pytest's
exit 5 (
no tests collected) and returns 0 — written for the legitimate "fewermodules than shards" case. Combined with a collapse, eleven of twelve SLM shards
would each exit 0 with a
::notice::, one shard would silently run the whole suite,and the aggregator would report that every shard passed.
That is worse than the exit-5 redness this repo has already been bitten by: it turns
the failure green instead of red.
It is also a regression against the old behaviour —
pytest-splitdegrades toequal-count contiguous chunks across all groups when durations are missing.
What Changed
Tie-break on a per-shard bucket count as well as load, incremented unconditionally,
so weightless buckets round-robin instead of piling up.
Verification
Each new test run against both algorithms:
Every one fails against the old tie-break, so none of them is decorative, and the
realistic distribution is untouched — this fixes the degenerate case without
perturbing the balanced one.
Model Used
Claude Opus 5