Repository navigation
bug(ci): co_change_test intermittently fails with a corrupt object DB in its own temp repo — 'Could not read <sha>' #14323
Description
Hit on an unrelated PR (#14308, which touches only autobot-slm-backend/), so recording it rather
than chasing it there.
Signature
autobot-backend/code_intelligence/co_change_test.py::test_the_minimum_count_alone_changes_the_result,
python-suite shard 2/12, under xdist worker popen-gw1:
GitCommandError: git ('log', '--no-merges', '--pretty=format:%x00%H', '--name-only')
exited 128 in /tmp/pytest-of-runner/pytest-0/popen-gw1/test_the_minimum_count_alone_c0:
error: Could not read 7dd1865e09e511b309511064c3288e711141cd49
fatal: Failed to traverse parents of commit 42edf5d32a5968eff2c037cfac4e3ac4fb527cb3
This is not a timeout or a network blip. Git is reporting a missing object in the fixture's own
temp repository: a commit exists, its parent does not. The object database the test built is
incomplete by the time it is read.
Why it is worth a ticket rather than a re-run
The same test area has taken four environment-isolation fixes in five days:
| commit | issue | change |
|---|---|---|
7647e8ea6 |
#13882 | strip every inherited GIT_ variable instead of a list of nine |
da8a45993 |
#14114/#14117 | a failed git invocation is now distinguishable from empty history |
25e776b66 |
#13948/#13949 | isolate git from the ambient environment so -C is authoritative |
7b835d635 |
#13983 | (docs) |
Each addressed how the miner talks to git. None addressed whether the repository the fixture
builds is intact when read. "Could not read <sha>" is a different failure than the ambient-env
ones those fixed, and it survived all of them.
Where to look
pytest-of-runner/pytest-0/is shared across xdist workers, withpopen-gw1only a subdirectory.
Worth confirming the fixture's repo path is genuinely per-worker and that nothing prunes or
garbage-collects a sibling worker's objects.- Whether the fixture creates commits through a code path that can return before the objects are
durable, so a subsequentgit lograces the write. - Whether
da8a45993's "distinguishable from empty history" change means a partially-readable
history now raises where it previously degraded — in which case the corruption predates that
commit and was simply invisible.
Not reproduced on demand
Two other PRs' pipelines at the same time (#14302, #14309) had zero failing jobs, so it is
intermittent. That is what makes it worth the issue: an intermittent corruption is the kind that
gets re-run away until it fails on something that matters.
Activity
mrveiss commentedon Aug 16, 2026
Three of the suspects are refuted with evidence — recording so they are not re-investigated
Work done under #14376. That PR is hardening, not a fix, and does not close this issue. What it did establish is worth keeping independently of it.
Refuted: a temp repo shared across xdist workers
pytest-xdist's controller passes each worker an explicitbasetemp(xdist/workermanage.py:340-341).TempPathFactory.getbasetemp()therefore takes the given-basetemp branch —rm_rf+mkdironce, memoized for the worker's life — which bypasses pytest's numbered-dir retention and lock machinery (make_numbered_dir_with_cleanup) entirely. That machinery only applies to a controller's own, un-given basetemp. Two workers' trees are disjoint subtrees by construction; neither worker's cleanup can reach the other's directory.Refuted: a prior CI job's retention sweep reaching a live directory
This was the #14114 mechanism, and it was specific to a persistent self-hosted runner where
/tmpoutlives a job, so thepytest-Ncounter kept climbing and old numbers came due while a concurrent job still held them.ci.yml'spython-shardjob now runs onubuntu-latest— a fresh VM per job — so there is no prior job's/tmpto reach. Closed by infrastructure already in place before this issue was filed.Refuted:
git gc --autopruning objects mid-testThe fixture's repo was measured at 78 objects, 0 packs.
git gc --autoproceeds only whenneed_to_gc()is true —too_many_loose_objects()(gc.auto, default 6700) ortoo_many_packs()(gc.autoPackLimit, default 50). At 78 and 0, neither can be true, so auto-gc never runs in this fixture.This also rules out the
gc.autoDetachvariant:autoDetachgoverns only how an auto-gc that has already decided to run behaves. If the decision is never reached, the setting is never consulted. #14376 setsgc.auto=0/gc.autoDetach=falseanyway as fixture hygiene, but by this measurement that cannot be what was corrupting the store, and the tests it adds assert the config values rather than the absence of corruption — they stay green if this recurs.Not applicable: tmp_path reuse or a copied
.gitmktemp(numbered=True)never reuses a name, and nothing copies a.gitdirectory.What still stands
The failure is
Could not read <sha>on a parent commit duringgit log --no-merges --name-onlytraversal — an object written and then unreadable, with no housekeeping in play. Suggested next lines of enquiry, in order:subprocess.runwrites are fully visible to the followinggit logunder the CI filesystem (overlayfs in a container,core.fsyncObjectFilesdefaults). This is the "returns before the objects are durable" wording, but at the filesystem layer rather than the housekeeping layer.git count-objects -v,git fsck --connectivity-only, and the.git/objectslisting from the failing temp repo. Any of the three would separate "never written" from "written then lost".Keeping open — the mechanism is not identified.
fix(ci): stop co-change test fixtures racing git's own background hou…