Repository navigation
test(ci): unrelated tests fail inside python-suite shards when the split shifts — two PRs blocked by defects they cannot reproduce #14111
Description
Activity
Narrowing: the polluter is outside the failing test's own directory
The whole
code_intelligence/directory passes locally, together, in one session:$ python3 -m pytest autobot-backend/code_intelligence/ -q 755 passed, 1 skipped in 205.90sSo the interaction is not within that package — it comes from another directory that lands in the same shard. That matches the
sys.modulesleak already reported in the other shard, where a stub installed bytests/orchestration/was still live duringtests/resiliencecollection.Reproducing a shard locally — the missing acceptance criterion
The exact CI invocation is
.github/workflows/ci.yml:258-266; substituting the shard number reproduces one shard's composition and ordering:./venv/bin/python -m pytest \ autobot-backend autobot_shared autobot-tts-worker repo_tests tools scripts pipeline-scripts \ -n auto --dist loadscope \ --splits 12 --group <N> \ --durations-path .test_durations \ -m 'not integration and not slow and not distributed and not performance' \ --tb=short
Two things that matter for anyone using this:
- Use
./venv/bin/python, notpython3. The venv is 3.14.6 and matches CI; the system interpreter is 3.10 and produces phantom failures. .test_durationsmust be present — it is what pytest-split chunks against, so a stale or missing file changes the composition and the reproduction with it. The committed copy is dated 2026-08-03, which is itself worth noting: shard composition drifts as the suite grows away from that snapshot, independently of any PR.
That last point may be the actual mechanism.
duration_based_chunkscuts the collected list into contiguous chunks weighted by recorded durations. Tests added since the snapshot have no recorded duration, so they shift boundaries in ways nobody controls — which explains why two unrelated PRs, adding tests in unrelated places, each moved a different state-sensitive test into a shard that breaks it.Running shard 2/12 locally now to confirm; will report whether
co_change_test.pyreproduces.- Use
Correction: it is NOT shard ordering. My earlier hypothesis is disproved.
I reproduced CI's shard 2/12 locally with the exact invocation and the committed
.test_durations:LOCAL shard 2/12: 1898 passed, 1 skipped, 1 error (unrelated ImportError), 649s CI shard 2/12: 1 failed, 1897 passed, 3 skipped, 125sSame total (1898), same composition, same order — and it passes locally. So shard placement and the stale durations file are not the cause, and my "boundaries shift as the suite grows" theory above is wrong. Withdrawing it rather than leaving it to be read as the explanation.
The variable is the environment, not the split.
Better hypothesis, with a documented precedent in the same file
co_change_test.pyhas a known history of exactly this failure mode._hermetic_git_env(:41-59) exists because of #13983:with
GIT_INDEX_FILEexported, every xdist worker stages into one index while committing in its own tmpdir repo, so a worker commits a tree whose blobs live in another worker's object store … That reads as repository corruption and is nowhere near the code under test.assert 0 == 1— the analysis finding no coupled pair in a repo purpose-built to contain exactly one — is that shape: the commits did not land as the fixture intended.The strip list (
_INHERITED_GIT_VARS,:28-38) is thorough andGIT_CONFIG_GLOBAL/SYSTEMare pointed at/dev/null, so the obvious leaks are closed. What differs between the two runs is worker count:-n autoon the CI runner vs this machine.--dist loadscopekeeps the grouping stable but not the concurrency, so a residual cross-worker interaction would surface at one width and not the other. That would also explain why it is intermittent rather than deterministic.What is established vs still open
Established:
- neither PR's diff touches the failing code;
- the test passes alone, in its own directory (755 passed), on base, and in the exact CI shard composition locally;
- the file has a documented xdist/git-corruption history whose signature matches this failure.
Still open: the precise trigger on the runner. I have not reproduced it, so I am not claiming the worker-count explanation is proven — it is the best-supported hypothesis, not a conclusion.
Suggested next step for whoever picks this up
Re-run the shard locally at CI's worker count (
-n <runners_cpu>instead ofauto), and if it reproduces, bisect the remaining shared state between workers. If it does not, capture the failing runner'sgit --versionand fullGIT_*environment — the runner configures git credentials viaincludeIf.gitdirdirectives, which the hermetic env does not currently neutralise for--localscope.The promtool instance on #14061 may or may not share this cause; it should be checked separately rather than assumed to be the same bug.
Second correction: my reproduction was run on the wrong tree
The "same composition, passes locally" comparison above is invalid. I ran the shard reproduction from the main tree, on
Dev_new_gui— not from the PR branch. So I compared base's shard 2/12 against the branch's shard 2/12 and reported them as the same thing.Since the branch adds test files, its collected set differs, so its shard boundaries can differ. That is exactly the mechanism I then declared disproved.
Withdrawing the "it is environment, not ordering" conclusion. Both hypotheses are open again.
What is now established, and it is stronger than a flake
co_change_test.py::test_the_minimum_count_is_inclusive_at_its_boundary — assert 0 == 1has now failed twice on PR #13991, both times in shard 2/12:run 31539731341 1 failed, 1897 passed run 31563090912 1 failed, 1896 passedSame test, same assertion, same shard, two independent runs. That is reproducible on CI, not intermittent, which also means "re-run until green" would not have worked here — worth noting given how tempting that was.
The differing pass counts (1897 vs 1896) are their own signal: the shard's contents are not stable between runs even on one branch.
Correct experiment, now running
The same invocation, from the branch worktree:
cd .worktrees/issue-13919 ../../venv/bin/python -m pytest \ autobot-backend autobot_shared autobot-tts-worker repo_tests tools scripts pipeline-scripts \ -n auto --dist loadscope --splits 12 --group 2 \ --durations-path .test_durations \ -m 'not integration and not slow and not distributed and not performance'
If it reproduces, the bug is debuggable locally and the ordering hypothesis is back. If it does not, the difference is genuinely the runner environment and the next step is capturing its
git --versionandGIT_*state.Recording the methodological error too, because it is the more useful lesson: a shard reproduction has to run on the branch under test. Running it from the main tree answers a different question and looks identical while doing so.
Root cause found: git repository corruption under xdist, not the split
The failing shard's log carries the answer. It is not ordering, not the durations file, and not my earlier "environment" hand-wave:
WARNING code_evolution_miner.py:145 git ('log', '--no-merges', '--pretty=format:%x00%H', '--name-only') exited 128 in /tmp/pytest-of-runner/pytest-0/popen-gw1/test_the_minimum_count_is_incl0: error: Could not read 1f6e052928008386261b61031e626f1ba3db2876git logexits 128,_run_gitswallows it to"", the analyzer sees no commits, and the test reportsassert 0 == 1. The assertion is a symptom; the defect is that the fixture's throwaway repo is corrupt — a commit exists whose object cannot be read from that repo's own object store.Note
popen-gw1: an xdist worker.This is #13983's bug, not a new one
co_change_test.py:41-54documents the same class, fixed once already:with
GIT_INDEX_FILEexported, every xdist worker stages into one index while committing in its own tmpdir repo, so a worker commits a tree whose blobs live in another worker's object store:
error: invalid object 100644 <sha> for 'solo_12.py'
That reads as repository corruption and is nowhere near the code under test.Same mechanism, different message —
Could not read <commit-sha>rather thaninvalid object <blob-sha>. So #13983's fix is incomplete: something still crosses workers after_hermetic_git_envclosed theGIT_INDEX_FILEpath.Determinism, and why re-running would not have helped
Two independent runs on PR #13991, same test, same shard:
run 31539731341 1 failed, 1897 passed run 31563090912 1 failed, 1896 passedReproducible on CI, never locally — I ran the full shard from the branch worktree twice (1929 passed) with
co_change_test.pyconfirmed present in the local shard. So this needs the runner's concurrency, not its ordering.A real inconsistency found while tracing it (not the cause)
The two strip lists have drifted:
var co_change_test._INHERITED_GIT_VARScode_evolution_miner._REPO_OVERRIDING_GIT_VARSGIT_CEILING_DIRECTORIESstripped not stripped GIT_CONFIGstripped not stripped The test hardens its own git calls more than the production helper hardens its. Neither explains
Could not read, but one SSOT list would stop them diverging further.Suggested next step
The corruption happens at commit time in the fixture, so instrument there rather than at the read: after
_commit, rungit -C <repo> fsck --no-progressand fail loudly with the worker id. That turns "the analyzer found nothing" into "worker gwN wrote an unreadable object", which is the fact #13983 needed and did not have.Bearing on the blocked PRs
Neither #13991 nor #14061 touches this code. This is pre-existing test-infrastructure corruption that a shard reshuffle exposes, so it will keep blocking arbitrary PRs until fixed — and each author will see a different test fail in a different shard, which is why it has read as flakiness rather than one bug.
The strip-list drift is fixed, and better than I proposed
I was about to consolidate the two hand-written
GIT_*denylists into one SSOT. Another session already did it in #13882, and inverted the approach rather than merging the lists:$ git show origin/Dev_new_gui:autobot-backend/code_intelligence/code_evolution_miner.py | grep startswith 116: return {k: v for k, v in os.environ.items() if not k.startswith("GIT_")}Its own reasoning is the part worth keeping:
Both are denylists — narrower than their own subject, correct for the failures already seen and silently wrong for the next one. Git has more than nine such variables and gains new ones between releases, so neither list could ever be finished, only extended once per incident. Inverted here: strip everything beginning with
GIT_.That is a strictly better answer than the one I had planned. Backing off; my worktree is disposed and nothing of mine is in flight for this issue.
Does it close this issue?
Plausibly, and I am not claiming it. The failure I root-caused was
git logexiting 128 witherror: Could not read <commit-sha>inside an xdist worker's tmpdir repo — object-store corruption, which a leakedGIT_*variable is a credible cause of. Stripping the whole prefix removes that entire class.What would settle it: the next PR that shifts the shard split. If
co_change_test.py::test_the_minimum_count_is_inclusive_at_its_boundarystops failing on CI, this is closed by #13882. If it recurs, the leak was never the mechanism and thefsck-after-commit instrumentation suggested above is the next step.Worth noting for whoever closes this: my two blocked PRs (#14237, #14247) are now green on the same suite, which is weak positive evidence but not proof — they may simply not be landing
co_change_test.pyin a hostile shard.The second instance on #14061 (
promtool_rules_test.py) was never traced to the same cause and should not be assumed closed by this.Third instance — same file, same shard, different test (PR #14249)
PR diff shard failure #14249 chat_history/+env_registry.py2/12 code_intelligence/co_change_test.py::test_dividing_by_the_smaller_count_would_have_reported_it—GitCommandError: git ('log','--no-merges','--pretty=format:%x00%H','--name-only') exited 128 … error: Could not read a9bac6ea794a87a14c93b7dd1215a5318ad9167fSame file and same shard as the #13991 instance already recorded here, a different test within it, and a different failure mode — an unreadable git object rather than an assertion mismatch. The diff touches nothing in
code_intelligence/. It surfaced immediately after merging base into the branch, which is what reshuffled the split.Evidence it is not the PR
# in the PR's worktree, at its own HEAD pytest autobot-backend/code_intelligence/co_change_test.py -q → 40 passed pytest autobot-backend/code_intelligence/ -q -n 4 → 778 passed, 1 skippedBase was green at
b9dcc580(AutoBot CI/CD Pipeline, success) at the time.One hypothesis ruled out, so nobody re-checks it
The obvious suspect was
GIT_*environment leakage across xdist workers — the #13983 failure mode this very file was hardened against, and the error text is the same family ("Could not read " = an object living in another worker's store).It is not that, in either direction:
- The production path is already hermetic.
code_intelligence/code_evolution_miner.py:116strips everyGIT_-prefixed variable before invoking git, so the miner cannot inherit a leaked one. - Nothing in the repo leaks one.
grep -rn 'environ\[.GIT_|putenv(.GIT_|environ.setdefault(.GIT_' --include=*.py(excludingvenv/) returns hits in onlyco_change_test.pyitself, and every one is wrapped intry/finallycleanup. No conftest touchesGIT_.
So the surviving explanations are shard-neighbour state pollution (this issue's thesis) or runner-level tmpdir/object-store interference — not environment inheritance.
Suggested addition to this issue's scope
The three recorded instances are all
assert-style or subprocess-level failures inside throwaway git repos incode_intelligence/. Worth checking whether that directory should be pinned to its own shard group, or its repo-building fixtures moved offtmp_pathonto a worker-unique root — a narrower fix than making the whole suite order-independent, and it addresses two of the three instances directly.- The production path is already hermetic.
Negative result: the runner's layout is reproducible locally, the failure is not
Following up my third-instance report above. I tried to reproduce PR #14249's shard-2/12 failure locally by matching the runner's configuration exactly, at the same commit the failing run used (
040f15ad7).Attempt Configuration Result 1 code_intelligence/alone778 passed, 1 skipped 2 Exact shard command, -n auto(22 workers here)2093 passed, 3 skipped 3 Exact shard command, -n 2— matching the runner's2 workers [1991 items]2093 passed, 3 skipped Attempt 3 is the one worth recording.
--dist loadscopegroups by module, so worker count determines which modules share a process — the obvious mechanism forsys.modulespollution to reach a victim. Matching it changes nothing locally.What that leaves
Two hypotheses are now ruled out for this instance —
GIT_*inheritance (see previous comment) and worker-count-driven scope grouping. What remains is something about the runner environment itself rather than the test schedule: the leak guard reports the pollution as known and baselined locally (code_intelligence/conftest.py, 1 key, onrepo_tests/sys_modules_leak_baseline.txt), yet the same pollution coincides with a hard failure on the runner.An unexplained discrepancy worth someone's attention
The runner collects 1991 items for group 2; the identical command at the identical commit collects 2096 here. Same
.test_durations, same seven paths, same--splits 12 --group 2. Roughly 105 items differ, and until that is explained, no local run is testing the same set the runner is — which would make every "passes locally" claim on this issue, including the ones in the issue body, weaker than it looks.That discrepancy seems more tractable than the pollution itself, and it gates the rest: it is hard to debug shard-neighbour effects when the shard's membership cannot be reproduced. Suggest starting there.
Cross-ref: PR #14249 is labelled
blockedwith this evidence; it is not merging on a red check.Root cause of the reshuffling, and of why it cannot be reproduced locally
Both symptoms this issue describes come from one mechanism. Evidence below is against
origin/Dev_new_gui.The measurements
total collected, the 7 CI paths, locally 25,982 tests entries in .test_durations 22,088 tests with NO recorded duration 3,894 (15%) group 2 items, runner 1,991 group 2 items, same command + same commit 2,096The mechanism
.github/workflows/ci.yml:239-243documents the algorithm: pytest-split'sduration_based_chunkscuts the collected list into contiguous chunks, chosen deliberately so whole modules stay in one shard (theleast_durationalternative splits modules and was rejected).Contiguous chunking means a shard's membership is a pure function of:
- the collected list — its order and its membership, and
- the per-test durations used to place the cut points.
Both inputs are unstable:
- Adding or removing any test file shifts every downstream boundary. Not just the shard the file lands in — every later shard gets a different set of neighbours. This is exactly what this issue observes: "adding tests shifts the pytest-split boundaries, and whatever lands next to a state-sensitive test fails." The trigger is mechanical and affects PRs whose diff is nowhere near the failure.
- 15% of tests have no recorded duration, so their placement rests on pytest-split's fallback rather than a measurement, making the cut points softer than they look.
Why local reproduction fails — and why the issue body's "passes locally" evidence is weaker than it appears
A module whose optional dependency is absent raises
Skippedat import, so its tests are never collected. This checkout has 23 of 67 requirements absent. A different collected set produces different contiguous boundaries, so--splits 12 --group 2does not name the same tests locally as it does on the runner — 2,096 vs 1,991 here.That is the 105-item gap I flagged earlier, and it is not incidental. It means:
- No local run of a shard is testing what the runner tested, so "passes locally" cannot clear a shard failure.
- Shard membership varies with which optional dependencies happen to be installed, so the same commit shards differently in different environments.
I could not reproduce PR #14249's failure across three matched configurations (
-n auto,-n 2matching the runner's2 workers, and the package alone). That was never going to work: the configurations matched, the input set did not.What would actually fix it
Assign whole modules to groups by a function of the module path plus its recorded duration — greedy bin-packing over the sorted module list — instead of cutting a contiguous slice of the collected list.
- Preserves the property the current choice exists for: modules stay whole,
--dist loadscopekeeps behaving as it does today. - Adding a test file then perturbs at most its own module's placement and whatever spills after it, instead of re-dealing every shard.
- Membership stops depending on collection order, so a developer can reproduce a shard even with a partial dependency set.
This does not contradict the "Do not switch it" note at
ci.yml:243; that warns againstleast_duration, which scatters individual tests across shards. A module-level stable assignment keeps modules intact.Fixing the assignment removes the trigger. The underlying cross-test pollution (five
_SysModulesLeakWarnings in the #14249 run, one fromcode_intelligence/conftest.py) is a separate defect and stays worth fixing — but it only bites when the shuffle puts a polluter next to a victim, which is what this change stops.I have not implemented this: it is a CI-wide change sitting next to an explicit do-not-change note, so it wants a decision rather than a surprise PR.
Two updates: a real cause ruled out, and stranded WIP recovered
#14334's red shard was NOT this issue. I checked before assuming, and it is worth recording because it is the exact shape that gets waved through as "the known shard flakiness":
autobot-slm-backend/slm/agent/health_collector_state_change_test.py:192 → autobot-backend/services/notification_service.py:49 E ModuleNotFoundError: No module named 'services.gateway'That was a genuine regression from #14334 itself. The slm test loads a backend module by path and stubs its imports in
sys.modules; #14334 added a new top-level import over in the backend, which the stub list did not cover. Under the slm rootdirservicesresolves to the slm package, so the backend'sservices.gatewayis genuinely absent.Reproduced locally by running with the slm rootdir (
cd autobot-slm-backend && pytest …) — same error, same two tests — then fixed and re-run: 14 passed. It never reproduced from the repo root, which is why it looked CI-only.The lesson generalises to this issue: a cross-tree loader that stubs its dependencies fails with a bare
ModuleNotFoundErrornaming a module that exists perfectly well in its own tree. That reads as a broken checkout or a shard artefact, not as a missing stub — so it will be misattributed here again.Stranded WIP recovered
Found uncommitted and untracked in an unrelated worktree during session close: a module-level stable-shard implementation plus its tests, written earlier and left behind. Now committed on
issue-14111-stable-shard(14a8f5d60) with a handoff, so it survives a worktree sweep.It does not work yet, and its own tests say so:
test_adding_a_module_moves_no_existing_module AssertionError: 271 existing modules changed shard when one module was addedAll four
TestStabilitycases fail the same way — contiguous chunking still reshuffles on any change. The 12 other tests pass. A stable assignment has to be a function of module identity (hash bucket, or a persisted map), not of position in a sorted list. The failing tests are the specification for that redesign.No PR opened — it would be red with no fix. Not dead code either: unfinished work against this issue.
Measurements for whoever is on the stable-shard branch
A session is working this in
issue-14111-stable-shard(the wip commit says the implementation does not yet satisfy its tests). Handing over what I measured so it is not re-derived. I am not touching that branch.Algorithm comparison, computed from the real
.test_durations(1,283 modules)Algorithm Balance by test count Modules moved when ONE module is added contiguous (current) — re-deals every downstream shard LPT greedy over modules 1.15x 1,019 of 1,283 pure hash → shard 2.97x 0 hash → 512 buckets → LPT table 1.18x 0 LPT looks like the obvious choice on balance and is the worst possible choice here: it is as unstable as what it replaces. Pure hash is perfectly stable but leaves one shard carrying ~3x the tests of another, and the slowest shard gates the job. The bucket table gets both.
Stability was also 0 for growing an existing module and for removing one, across 50 trials each.
The distinction that is easy to get wrong
The durations file and the collected set are different inputs, and only the first may feed the table.
Build the table from
load_module_weights(".test_durations"). A PR that adds tests changes what is collected; it does not change.test_durations. That is precisely why the assignment stays stable.If the table is instead built from the modules that were just collected, every stability property disappears — adding a module changes the weights, which changes the table, which re-deals everything. My first version of the tests made exactly this mistake and failed loudly on all four stability assertions, which is how I found it. Given the wip commit message, this may be the same gap.
Balance should be computed on test count, not recorded seconds
.test_durationssums to 338 seconds across 1,283 modules, while a shard takes minutes. Recorded duration captures only per-test call time — import and collection dominate real shard cost. Balancing on that signal optimises the wrong quantity, which is a second, independent problem with the current contiguous-by-duration approach.For reference, the same bucket table balanced on recorded seconds instead gives a 23.9x spread — i.e. the two objectives disagree sharply, and test count is the one that tracks reality here.
Properties worth asserting
- adding / growing / removing a module moves no other module (the point of the change);
- a partial checkout — 23 of 67 optional deps absent here — assigns identically to a full one, which is what makes a shard reproducible locally at all;
- whole modules never split across shards, so
--dist loadscopeis unaffected; - heaviest shard within a bounded factor of the lightest, asserted, so a later bucketing change cannot quietly unbalance CI;
- regenerating durations is allowed to re-deal — name it in a test so it is a known, deliberate operation rather than a surprise.
Both halves are up, as a stack
- test(ci): stable module-level shard assignment (#14111) #14646 — the assignment and its tests. Inert: nothing passes
--shard-splitsyet. - ci(shards): adopt the stable module assignment for both suites (#14111) #14648 — wires it into
ci.ymlfor both suites, stacked on test(ci): stable module-level shard assignment (#14111) #14646.
Split so the
ci.ymlchange gets its own review. It edits a note saying "Do not switch it", which it does not violate — that warns againstleast_duration, which scatters individual tests and splits modules, and a module-level assignment is exactly what keeps them whole. The note is rewritten to say what is now true.The measurement that closed this
Thanks to the session that posted the algorithm comparison — it saved re-deriving it, and named the trap I had already fallen into. My stability tests rebuilt the table from the mutated collection on both sides of each comparison, so they asserted a property the design deliberately does not have. The tests were wrong, not the implementation. Recorded in the file so it is not repeated a third time.
Reproduced the posted figures independently: 1282 modules, 1.18x spread by test count, zero modules moved across 50 trials each of add/grow/remove.
The verification I would want if I were reviewing this
Assignment being stable is not enough — it also has to be weighted, and those fail differently. If pytest's node-ID module paths did not match the durations-file keys, every module would hash consistently (still stable) while every weight was silently ignored, and balance would degrade with nothing failing.
So the check is that the plugin selects exactly what the offline table predicts. For
autobot-backend/a2a: predicted groups 6/7/8/11 at 33/46/53/48 rows; the real plugin under real pytest selected 0 for group 1, 0 for group 4, and 46 for group 7.Also, directly answering #13637 — no empty shards on either real target set:
backend: 1282 modules, 22087 tests, 1.18x, empty: none slm: 106 modules, 1501 tests, 1.22x, empty: noneThat note said "fixing the packing is the real repair"; #14648 updates it rather than leaving the claim stale, while keeping the exit-5 tolerance — the condition it guards is a property of the module set, not the algorithm.
Still unmeasured: wall-clock shard times under the new distribution. Balance is asserted on test count, a better proxy than recorded seconds but still a proxy. The first CI run on #14648 is that measurement.
- test(ci): stable module-level shard assignment (#14111) #14646 — the assignment and its tests. Inert: nothing passes
- added 4 commits that reference this issue
on Aug 21, 2026 Closed — the stack landed
- test(ci): stable module-level shard assignment (#14111) #14646 (
f0f3...) — the assignment and its tests - ci(shards): adopt the stable module assignment for both suites (#14111) #14648 (
45596cf87) — wired intoci.ymlfor both suites
Verified in base:
$ git show origin/Dev_new_gui:.github/workflows/ci.yml | grep -c 'shard-splits 12' → 2 (both invocations) $ git show origin/Dev_new_gui:.github/workflows/ci.yml | grep -c -- '--splits 12' → 0 (pytest-split gone) $ git show origin/Dev_new_gui:repo_tests/stable_shard.py | grep -c 'def ' → 7All 12 shards green on the merge commit, including shard 9 — the one that failed twice on the way here.
What the failures taught, since they are the useful part
Shard 9 failed twice with zero test failures and exit code 1: the sys.modules leak guard, on 8
plugin_sdk*keys.Those keys are not a stray stub.
autobot_shared/plugin_sdk/loader.pydeliberately aliases the canonical package under its top-level name (#11636) so a plugin importingplugin_sdk.basegets the same singletons. That is a process-global mutation, and it is exactly why my first fix could not work: it filtered against a per-testbeforesnapshot, so whichever test loaded a plugin first in the shard installed the aliases and every later test — including the one being blamed — saw them as pre-existing and left them.Changing the shard assignment did not create that leak; it changed which files share a shard, so the pairing became visible. The fix removes the aliases unconditionally, which is safe because the loader recreates them on demand and the backend-local
plugin_sdkis a re-export shim — the identity invarianttests/test_plugin_sdk_shim.pypins still holds.Worth recording honestly: I could not reproduce this locally at any point, including a full local run of shard 9, and a mutation test confirmed my reproduction could not tell the old and new implementations apart. The fix was reasoned from the mechanism rather than demonstrated against the failure, and CI was the only real verification. A clean local run proved nothing here.
The measurement, for whoever tunes this next
backend: 1282 modules, 22087 tests, 1.18x spread, no empty shards slm: 106 modules, 1501 tests, 1.22x spread, no empty shardsThe slm figure is the direct answer to #13637, which measured group 1 taking 497 of 1644 while groups 8-12 took none. That note said "fixing the packing is the real repair" — this is it, and the exit-5 tolerance is deliberately kept, since the condition it guards is a property of the module set rather than the algorithm.
Still unmeasured: wall-clock shard times under the new distribution. Balance is asserted on test count, which tracks reality better than recorded seconds but is still a proxy.
- test(ci): stable module-level shard assignment (#14111) #14646 (
Two PRs whose diffs touch none of the failing code were each blocked by a different unrelated test failing inside a
python-suiteshard. Both pass locally, in isolation, and onDev_new_gui. Adding tests shifts the pytest-split boundaries, and whatever lands next to a state-sensitive test fails.The two instances
test_skill_distillation_scheduler.pyrepo_tests/promtool_rules_test.py::TestAMissingPromtoolIsLoudOnCI::test_a_developer_machine_without_promtool_still_skips—assert 1 == 0chat_workflow/code_intelligence/co_change_test.py::test_the_minimum_count_is_inclusive_at_its_boundary—assert 0 == 1Each shard reported one failure among ~1300–1900 passes, and the second red check in each case is just the
python-suiteaggregator ("Fail if any shard failed"), not a second defect.Evidence they are not caused by the PRs
Neither PR's diff is anywhere near the failing file.
Why shard placement is the suspect
The shard that failed on #14061 was already reporting cross-test pollution in the same run:
So at least one shard is executing with a module-level stub left installed by an earlier test. The leak guard already detects this and warns; nothing fails on it.
co_change_test.py'srepofixture is function-scoped and builds its own git repo, so it is self-contained — which makesassert 0 == 1(the analysis found no coupled pair in a repo purpose-built to contain one) point at environment or leaked global state rather than fixture sharing. That area has had recent churn (#13832/#13855 ported the history crawler to the git binary, plus #13980/#14014), so a git-version or config dependence on the runner is a live hypothesis.Why this matters
The failures are not reproducible by the PR author, land in code the author never touched, and move as the split moves. The practical effect is that any PR adding tests can be blocked by an unrelated defect it cannot debug, and "re-run until green" becomes the path of least resistance — which is how a genuinely order-dependent test becomes permanent.
Relevant existing context: #13162 (the python suite is not a required check) and the known pytest-split failure mode where later shards go empty.
Acceptance criteria
co_change_test.py's failure is root-caused — environment (git binary/version) vs leaked state — rather than re-run awaysys.modulesstub leaking out oftest_causal_error_recovery.pyis scoped to the nodes that need it (the guard already names it)Related
#13162 · #13927 (added the promtool test) · #13832 / #13855 (crawler ported to the git binary) · PRs #13991, #14061