Repository navigation
Flaky shell tests fail native-shell-windows CI intermittently #754
Description
Symptom. Two timing-based tests in CoreVideoPro.MediaCore.Tests each failed once in CI on PRs that touch nothing near them, and passed on rerun / locally (2,287 of 2,287):
MediaCoreHandshakeTests.IncompatibleRequestOnlyChildIsTerminallyRejected—Assert.Throws() Failure: Exception type was not an exact match(PR Remove the orphaned React prototype and Node core simulator (#738) #745, run 36952291653, 2026-10-02).CoreRecoveryRejoinTests.AStartDuringCrashRecoveryStillRejoinsZoom—Assert.Single() Failure: The collection contained 2 itemsafter 15 s (PR Protocol manifests: cross-language parity tests (#747) #753, run 37054430273, 2026-10-02).
Not confirmed whether these are test races or real races in the supervisor that only a slow runner exposes. The second one is in the crash-recovery rejoin path, so it is worth ruling out a real double-rejoin.
Done when. Each test is either made deterministic or the underlying race is fixed, with the cause stated.
Activity
iamfatness commentedon Oct 2, 2026
Third intermittent failure in the same job:
scripts/test-audio-meter-stability.ps1—Meter probe failed or crashed: exit=1 … InvalidOperationException: Probe meters did not load(PR #756, run 37061782651, 2026-10-02; that PR only moves C++ code between files). The same probe failed once on PR #727 on 2026-10-01 and passed on rerun.Split production module composition out of StubModules.cpp (#742) (#756)
iamfatness commentedon Oct 2, 2026
Another timing-based one, local this time:
FfmpegChildRetirement.SenderStopReapsItsActualTransportChildfailed once in the full native suite on the dev build ofmain4439535 ("restart did not launch a replacement child": the stand-in FFmpeg child did not write its marker within the test's 3 s deadline). Passed 3 of 3 on immediate rerun of the filter. The machine was otherwise idle apart from this session.iamfatness commentedon Oct 3, 2026
Tried to reproduce
CoreRecoveryRejoinTests.AStartDuringCrashRecoveryStillRejoinsZoomlocally on main 75f9ad2: 25 runs idle and 12 runs with six busy-loop processes pinning the CPU, 0 failures. Reading the supervisor:RecoverChildAsync(the only raiser ofRecoveryCompleted) is started exactly once perOnChildExited, guarded by_crashRecoveryOwed, andStartAsyncnever raises the event, so two outcomes would need two child exits or a second exit event for the same child. I could not produce either. The CI run that failed (37054430273) took 15 s for this test, which is theUntilAsyncdeadline, so a wait did hit its limit before the double outcome was counted. No code change; a real double-rejoin is neither confirmed nor ruled out. Next step when it recurs: capture the fake core's joins.log and the supervisor's media-core.log from the CI artifact.iamfatness commentedon Oct 3, 2026
Two more, both on commits that cannot have caused them:
SourceAudioIngress.MediaDemandUsesSteadyClockInsteadOfRenderTimestamponnative-metal-macosfor PR Bound the supervisor's stand-off for a self-managed retry and require it to be fresh (#612) #777 (run 37091659343, 2026-10-02): a 2 s deadline for the media decoder to deliver PCM, missed on the mac runner. Bound the supervisor's stand-off for a self-managed retry and require it to be fresh (#612) #777 changes only the supervisor stand-off policy.mainrun for the docs-only commit 75f9ad2 (BACKLOG: owner-accepted order of 2026-10-02 and the acceptance checklist #773) failed too (run 37074060203).Pattern so far: every intermittent failure is a wall-clock deadline (2 s, 3 s, 15 s) on a shared CI runner. The fix is likely one shape: drive these tests from an injected clock or a condition wait with a generous bound, not a fixed sleep budget.
Turn the deadline-based tests into bounded condition waits (#754) (#778)
iamfatness commentedon Oct 4, 2026
The same recovery assertion recurred on main 92e6b1e while preparing beta-2026-10-04-92e6b1e.
CI run: https://github.com/iamfatness/CoreVideoPro/actions/runs/37218167884 (attempt 1, native-shell-windows). CoreRecoveryRejoinTests.AStartDuringCrashRecoveryStillRejoinsZoom failed after 15 seconds: expected one recovery outcome, received ZoomRejoinFailed followed by ZoomRejoined with epoch 1:1:6164. MediaCore result: 2,293 passed, 1 failed.
The full 2,294-test MediaCore suite passed locally, followed by 10/10 isolated repeats of this case. The failure does not establish the underlying cause, and a real supervisor race is not ruled out. The hosted failed job is being rerun before beta publication.
Preserved the failed log and first-attempt windows-shell-test-results artifact privately under capture-reconnect-784/artifacts. The artifact contains only the MediaCore TRX; fake-core joins.log and supervisor media-core.log were not uploaded. Those remain the missing diagnostic evidence identified in the earlier comment. Reopening because the same assertion recurred despite the longer wait bound.
iamfatness commentedon Oct 7, 2026
October 6 recurrence during the #801 monitor-admission slice: full Release MediaCore suite reported 2 RecoveryCompleted outcomes in
CoreRecoveryRejoinTests.AStartDuringCrashRecoveryStillRejoinsZoom:(ZoomRejoinFailed, null)followed by(ZoomRejoined, "1:1:<pid>"), failing Assert.Single at line 87. 2,330 tests passed; this one failed. The native rebuild was running concurrently; that is context, not established attribution. No recovery/supervisor source changed in this slice.Private evidence retained locally:
artifacts/monitor-admission-full.logandartifacts/monitor-admission-support/monitor-admission-full.trx. No raw process logs or meeting material uploaded. A quiet targeted repeat follows the build. This matches the existing double-outcome symptom on #754; it is distinct from #798's initial roster request timeout.iamfatness commentedon Oct 7, 2026
Quiet repeat: the targeted recovery test passed, followed by the full Release MediaCore suite (2,331 passed). Original double-outcome failure remains retained; this does not establish its intermittent cause.
iamfatness commentedon Oct 7, 2026
PR #808 exact head 495b155 CI 37577966170, native-shell-windows job 112650998885, failed the handshake case already listed in this issue: MediaCoreHandshakeTests.IncompatibleRequestOnlyChildIsTerminallyRejected. 2,330 passed / 1 failed. Expected InvalidOperationException; actual OperationCanceledException during SendAsync's WriteLineAsync (MediaCoreSupervisor.cs:1695), before incompatible protocol rejection.
The existing fixture sets HandshakeRequestTimeoutMs=1000, RequestTimeoutMs=60000 and outer waits=90 s. HandshakeAsync passes HandshakeRequestTimeoutMs explicitly into SendAsync, which applies that value to its cancellation source for gate/write/flush/response. Thus the general 60-second request setting does not extend this explicit handshake's one-second budget; the fixture comment claiming it does is incorrect. StartAsync first waits the same one-second unsolicited-handshake grace, then issues that one-second request, explaining the roughly two-second failure. The log does not establish why the pipe write failed to progress within that budget.
This PR changes no supervisor or C# test code. The failed job/log is retained and an exact-head hosted shell-job repeat was requested. A repeat pass does not close either this handshake item or the separate duplicate RecoveryCompleted item. No production timeout or assertion was changed.
Failure: https://github.com/iamfatness/CoreVideoPro/actions/runs/37577966170/job/112650998885
iamfatness commentedon Oct 12, 2026
The same known failure recurred on PR #868 at 6486e35 in run 38200208312, native-shell-windows job 114651711616. CoreRecoveryRejoinTests.AStartDuringCrashRecoveryStillRejoinsZoom reported both ZoomRejoinFailed and ZoomRejoined; 2,381 other managed tests passed. Local full suite previously passed 2,382/2,382. This is retained as an unresolved recovery-path finding rather than weakening the assertion. CI log: https://github.com/iamfatness/CoreVideoPro/actions/runs/38200208312/job/114651711616