Repository navigation
Windows test suite remains dispatch-only until full suite is green #1059
Description
Client or integration
Other (test suite / CI)
Area
Windows CI / cross-platform test suite
Current status
The Windows full test-suite leg is currently dispatch-only and does not gate normal development or release promotion.
This issue tracks the Windows failure burn-down and the eventual restoration of Windows as a required CI gate.
The original issue description cited approximately 207 failures from an older aborted promotion run. That number is no longer a reliable baseline.
A newer full Windows dispatch against dev produced the current useful baseline:
| Shard | Result |
|---|---|
| 1/4 | timed out at the 15-minute cap; no final failure count |
| 2/4 | 8 failures |
| 3/4 | 21 failures |
| 4/4 | 47 failures |
So the current known baseline is:
76 failures across the three completed shards
+ unknown failures in shard 1 because the shard timed out
The Windows leg should remain dispatch-only until a full run completes successfully on current dev.
Current reference run
Windows dispatch:
https://github.com/lidge-jun/opencodex/actions/runs/31095755263
All four Windows shards failed or timed out in that run.
The old ~207 count from the earlier promotion run should not be used as the burn-down target.
Main failure family
The recoverable failures are not evenly distributed.
The largest coherent cluster is around the Codex config/catalog write substrate, including:
injectCodexConfigintegration;- Codex catalog sync hardening;
- Codex catalog restore;
- cross-process write-lock contention;
codex-journal.
The failure signatures point toward Windows-specific filesystem and process semantics around areas such as:
- atomic publication;
- replacement/rename behavior;
- permission normalization;
- file locking;
- concurrent writers;
- journal/CAS behavior.
These should be treated as a shared substrate problem where the evidence supports it rather than as dozens of unrelated test failures.
Remaining distinct failure classes
Other observed failures include areas such as:
- server same-target
429retry end-to-end behavior; - expired forward replay state related to Codex App resume silently drops prior context after the 1-hour replay-state TTL expires #702;
- Windows-specific process/test timing.
These should be handled separately once the dominant write-substrate failures are removed.
Shard 1 timeout
Shard 1/4 is a separate observability problem.
It exceeded the current 15-minute job limit before producing a usable final test result.
That means one quarter of the Windows suite currently provides no complete failure inventory.
Before treating shard 1 as simply another group of failing tests, determine:
- which tests consume the time;
- whether retries/filesystem waits are responsible;
- whether there is an actual hang;
- whether shard balance itself needs adjustment.
The objective is not merely to increase the timeout indefinitely. The shard needs to complete reliably enough to provide actionable CI signal.
Recommended burn-down order
1. Fix the shared config/catalog write substrate
Prioritize:
injectCodexConfig
catalog sync
catalog restore
atomic publication / permission semantics
2. Re-run the write-lock contention coverage
Determine whether those failures disappear with the same substrate correction or represent an independent Windows locking defect.
3. Fix journal behavior
Address remaining Windows-specific codex-journal failures after the underlying publication/locking behavior is stable.
4. Split unrelated tails
Handle genuinely independent failures such as:
server 429 retry
#702 expired forward replay
other isolated Windows-only failures
as separate focused changes where appropriate.
5. Resolve shard 1 completion
The complete four-shard suite must finish reliably before Windows can become a useful required gate.
PR expectations
Windows burn-down PRs should be small enough to review independently.
Each PR should identify:
- the failure family it addresses;
- the affected Windows shard(s);
- the before/after failure count for those tests;
- focused Windows validation;
- confirmation that Linux/macOS behavior is unchanged.
Do not combine unrelated Windows failures merely because they happen to appear in the same shard.
Gate restoration criteria
Windows can return to required CI only when all of the following are true:
- all four Windows shards finish within the configured CI budget;
- the full Windows suite is green on current
dev; - the result is repeatable rather than a single lucky run;
- Windows-specific fixes do not regress Linux or macOS;
- the workflow is changed from dispatch-only back to the intended required/gating behavior.
Until then, the Windows leg remains dispatch-only.
Historical context
The Windows suite accumulated failures because it was not previously exercised as a normal dev gate.
The important target now is not reproducing or preserving an old failure count.
The target is:
current dev
4/4 Windows shards complete
0 failures
Checks
- The original Windows regression is still actively tracked.
- A current full Windows dispatch has been measured.
- The obsolete
~207 failuresestimate has been removed as the baseline. - The dominant failure family has been identified.
- Shard 1 timeout is tracked as a distinct CI-signal problem.
- Four Windows shards complete successfully on current
dev. - Windows is restored as a required CI gate.
Activity
github-actions commentedon Aug 5, 2026
Automated translation bookkeeping — detected language: English.
fix(ci): drop the Windows leg from the release gate
lidge-jun commentedon Aug 6, 2026
Status: the Windows leg stays dispatch-only. Plan of record: burn down the ~207 failures shard by shard (management/server fixtures first, then platform process semantics), restore the gate only after a full green Windows run on
dev. Shard-scoped PRs welcome; each should name the shard and the failure class it eliminates.lidge-jun commentedon Aug 6, 2026
Fresh measurement. I dispatched the Windows leg against
devtoday — first run since 2026-08-01 — so the burn-down now has current numbers instead of the aborted promotion run's estimate.Run 31095755263, all four shards failed:
So 76 failures across the three shards that finished, plus an unknown count in the shard that never completed. The "~207" figure in the description came from the aborted promotion run and should be treated as unverified rather than a target.
The failures are not spread evenly — they cluster hard. Grouping the recoverable ones by suite:
Roughly three quarters are one problem family: Codex catalog and config file writing —
injectCodexConfig, catalog sync/restore, the write lock, and the journal. The failing assertions read like atomic-publication and path/permission semantics ("writes prepared bytes atomically with the right...", "a coordinator found group-readable is narrowed back to owner-only", "a native CAS with a matching generation but the wrong txId still conflicts"), which is exactly where POSIX assumptions leak:renameover an existing file,chmodbits, and file locking all differ on Windows.That changes the shard-by-shard plan for the better. Fixing the write-substrate family first should clear ~78 of the 76+ recoverable failures in one pass, rather than grinding through shards that are mostly symptoms of the same cause. Concretely, in order:
injectCodexConfig+ catalog sync/restore atomic publication (therename/permission layer they share)codex-journal, then the genuinely unrelated tails (429 retry, Codex App resume silently drops prior context after the 1-hour replay-state TTL expires #702 replay)Shard 1's timeout is its own defect and should be split out. A shard that cannot finish inside 15 minutes gives no signal at all, and the cap was already sized against a green suite — on Windows the retries and filesystem waits push it over. Either that shard needs a longer budget or the slow tests need identifying; until then we are blind on a quarter of the suite.
Raw failure names are in the run logs; I did not paste all 152 lines here.
Keeping this open. The plan of record stands, with the ordering above replacing "management/server fixtures first" — the measurement says the write substrate is the bigger and more coherent target.
docs(devlog): measure the Windows suite instead of restating the plan
[-]Windows test suite: ~207 failures, leg is dispatch-only until green[/-][+]Windows test suite remains dispatch-only until full suite is green[/+]fix(tests): stop asserting POSIX modes Windows cannot produce
lidge-jun commentedon Aug 18, 2026
Closing this tracker. The hosted Windows runners have been too unreliable for a meaningful burn-down (shard timeouts at the 15-minute cap, loaded shared runners distorting failure counts), so gating on them is not productive. The maintainer will run the Windows suite locally on real Windows hardware and push fixes from those runs instead; the
platform-windowsleg stays dispatch-only as an on-demand check. Shard-scoped fix PRs from local Windows runs are still welcome — name the failure class the PR eliminates. If hosted-runner reliability improves enough to restore Windows as a required gate, reopen this with a fresh full-run baseline.fix(ci): drop the Windows leg from the release gate
docs(devlog): archive the 48 units whose work landed on dev
feat(pao): land accumulated phases 20.20-20.63 implementation (pre-GO…