Repository navigation
Measure matched CPU p95 delivery-trace cost (#823) - #828
Merged
Merged
Conversation
iamfatness
force-pushed
the
codex/render-work-evidence-823
branch
from
October 8, 2026 01:07
3d79f8a to
879a287
Compare
iamfatness
changed the base branch from
codex/delivery-trace-aggregates-823
to
main
October 8, 2026 01:08
iamfatness
marked this pull request as ready for review
October 8, 2026 01:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#823: Mean and maximum CPU render-work counters cannot qualify the approved <1% p95 trace-instrumentation-cost gate. Add an independent, bounded opt-in QA histogram and connect the real worker's existing work duration to diagnostics and the maintained A/B harness. Media record adds one relaxed atomic increment only when explicitly enabled; no clock read, allocation, lock, file I/O or pixel work. Disabled instances allocate no histogram and retain unknown observations.
The read-only on/off judge subtracts fixed-warmup counts, rejects malformed/missing/reset/generation/overflow/accounting evidence, retains 1 microsecond histogram resolution and conservative endpoint-rank uncertainty, and compares traced p95 upper bounds against reference p95 lower bounds. It requires matching binary/source/judge/hardware/workload/flag manifests and successful output, recording and raw-trace evidence. PASS here cannot qualify GPU, source/content latency, receiver/display, A/V, real SDK or fleet. Normal logging and launch defaults remain unchanged.
Validation in progress: 35 portable tests pass (18 monitor/IPC, 11 per-frame trace, six timing/cost fault cases); generated contracts/diff checks pass. Production Release build, three native collector tests (including the real worker/snapshot consumer), ten trace regression tests and both trace-off GPU trials pass. PR #827 is merged; this PR is rebased onto main. Source 879a287, native tree 33b8f0a96035f59ddbd4bfc314991855548e90e8; exact manifest/source archive retained. Full native suite passes: 1,520 tests, zero failures (external network/NDI opt-ins not executed). All 17 checks pass on exact head 879a287, including native production/shell Windows, macOS Metal/stub, TSAN and CodeQL. Continues #823/#517 and closes neither; no beta/default or camera ABI change.
#823 / #517 qualification finding, retained without a PASS waiver
Matched same-binary timing experiment: source 879a287, native tree 33b8f0a96035f59ddbd4bfc314991855548e90e8, core SHA256 6C3715576855E064D3B056042ED415D774916D4227BF16DCC677371107BF7838. Timing collector enabled in both conditions; CPU preparation 1, GPU capture 0, buffer 2, eight fake 1080p30 I420 feeds, 1080p/1440p60 BGRA mappings, mixed Program at 1080p60, local recording; each inline/isolated trial measured 45 seconds after fixed 15-second warmup. Defaults/installed beta unchanged.
Trace-off trials passed output/recording/sample admission and timing-accounting checks. However the control already had substantial render tails: isolated mean 0.882 ms, p99 interval 11.749–12.730 ms, p99.9 interval 12.747–17.816 ms; two render completion overruns. Inline control had 64 overruns. These percentile intervals retain histogram/endpoint uncertainty and are not rounded point estimates.
Trace-on inline passed delivery/recording/raw trace. Trace-on isolated FAILED: five buffer underruns and five missing scheduled recording frames, four skipped CPU slots, nine render overruns; raw per-frame trace reports five misses/one gap with zero trace loss or export failures. Its high-percentile finite range was exceeded (>32.768 ms), so the timing distribution is INVALID rather than clamped to a healthy value. The matched cost comparison is INVALID because candidate delivery/recording failed; <1% trace overhead is not established.
The first divergence occurs 30.125–30.391 seconds into measurement. A 95.7085 ms CPU render-work interval precedes five misses for targets 2729–2733. Last timely Program submission 2728 occurred at 30.202053 seconds; next submission 2730 at 30.317059 seconds. Existing stage logs attribute this exceptional tick to Program 77.84 ms and monitor-request/build work 17.66 ms, not ingress (0.10 ms) or event drain (<0.114 ms maximum). The D3D renderer reports frame 2729: total 77.627 ms, encoder-export region 77.498 ms (historical
vcamtimer label); drawing 0.040 ms, upload 0.009 ms, readback zero. Independently scheduled shell and multiview exports simultaneously reported keyed-mutex AcquireSync scopes around 77.022/77.004 ms despite zero requested timeout. This locates a GPU-handoff/API-stall region; it does NOT prove which producer call, driver scheduling, OS preemption or tracing caused it. Some other periodic tails are attributed to ingress fetch and require separate attribution.No beta/default promotion or issue closure. Retain artifacts/render-work-823-first-{off,on}, raw captures, snapshots, stderr, exact manifest/source archive and artifacts/render-work-823-first-cost.json. Next focused diagnostic subdivision must distinguish producer mutex acquire, copy, release and remaining encoder-export work before changing the handoff. This evidence remains under the approved #823/#517 scope; it does not establish #825's separate wait-predicate amplification as the initiating cause.