Repository navigation
perf: Program render hits the one-frame budget at 16 participants with the 1080p wall #517
Description
Activity
- added a commit that references this issue
on Sep 14, 2026 Live validation 2026-09-21 confirms remaining render-budget pressure at a smaller mixed workload: RTX 4090, 9 Zoom participants, Speaker + Slides program, HEVC 1080p60/10 Mbps to YouTube. Installed native 66d5d62; no builds/heavy tests during measurement.
Over the clean interval from the 10:33 baseline to 10:38:52, CPU render deadline misses rose 79 -> 1328; skipped slots 12 -> 16. Over the final approximately 78 seconds, program-buffer underruns rose 57 -> 78 and overflows 41 -> 62. Logs report participant export averaging roughly 5-6 ms and peaking around 12 ms, with individual program CPU submissions reaching ~15 ms. This identifies a measured budget consumer, not proof of sole causation. GPU completion and final destination delivery remain unverified by those CPU counters.
Transport had no retries for 6m26s before a deliberate restart (which reproduced #569 queue overflow). YouTube played 1080p60 but had additional browser playback drops. Keep those separate from native render metrics. Evidence: artifacts/youtube-preparing-20260921/lifecycle-soak-5min.json and lifecycle-soak-before-restart.json. Full findings are on #569; this is not a smooth-output acceptance pass.
Implemented device reuse across participant resolution changes in fe9affb (PR #566 commit e0e6878). Live profiling found D3D11CreateDevice on the render thread every time Zoom switched 320x180 / 640x360 / 1280x720: 23-27 ms per resize. The exporter now retains its device/context and replaces only size-dependent resources. Ungraded BGRA exports also avoid an unnecessary intermediate CopyResource, and held-frame bookkeeping advances only after successful submission.
Five D3D export tests pass, including 12 alternating-size cycles that verify the same export device and actual new pixels read from a separate consumer device. In the installed 8-guest Zoom wall, 56 real resolution changes measured median 994 us, p95 1967 us, max 4187 us. This removes the recurring ~25 ms resize cost; initial device creation is still measured separately. No resolution/quality reduction was introduced.
This is a measured improvement, not closure of the full 16-guest performance gate. PR remains draft; full sustained-output acceptance is still being tested.
- added 5 commits that reference this issue
on Oct 3, 2026 Owner approved the delivery completion plan on October 3 and authorized execution, including #781 as the installation prerequisite. The approved plan is docs/reference/render-delivery-completion-plan.md. Execution begins with an isolated, controlled publication-contention experiment through Windows Frame Server; remaining scope includes installation, render completion, bounded diagnostics and final qualification. Backlog records the owner-approved dependency order. PR #780 remains draft and experimental defaults remain off pending evidence.
Controlled Windows experiment now demonstrates recovery of publication contention. Two explicit isolated DLLs use the same reader code with retry off/on and a separate QA mapping; the producer holds the seqlock odd for configured 8 ms windows while publishing rational 60 Hz identities. Actual publisher logs recorded zero source deadline overruns in both on runs and the repeated off control.
Retry off failed both controls; the repeated control recorded 30 repeats, 34 missing identities, 2389 slate/invalid markers and a 601.076 ms arrival gap. Retry on passed two measured minutes (3601 frames each, no repeats/missing/invalid/reordered); max intervals 18.306 and 20.588 ms. Retry was actually exercised: the second process logged 5461 attempted/recovered samples, zero held/slate emission. DLL counters include smoke/warmup and are distinct from measured receiver counts.
A 20 ms over-budget negative control failed and recorded 1800/1800 source deadline overruns; it cannot be called a healthy-source qualification. The judge now also rejects declared 60 fps with only 50 delivered samples/s. Scripts 160/160, existing retry 5/5 and SHM 10/10 regressions pass. Release diagnostic targets built. Exact loaded paths, hashes and all raw evidence are in preserved-local-evidence/camera-publication-qa/validation-report.md. Commit b3909f8 / PR #780.
This proves this specific race can be recovered through Windows, not the original show's sole cause or full hardware/latency/audio qualification. Consumer phases differ between trials and full matched latency is still open. Installed registration restored; production mapping untouched. Next approved dependency is #781 installation repair in a separate PR, followed by the remaining #517 render and diagnostic scope. An unrelated concrete Windows CI disposal failure is intake #782, not attributed to the camera patch. Current commit requires fresh CI.
21 remaining items
PR #815 merged as 15ad2ae. Its native tree 318242d04c58c05a1d0b6eaf2182b56dd32977b1 matches the tested candidate. All 1,496 native tests and 17 final-head CI checks passed.
The held-still/recreated-consumer test failed before repair and passed after it. The actual still worker rebuilds prepared BGRA views for a new compositor without changing CPU bytes, frame/epoch/observation identity or performing Program source uploads. Actual I420 consumer-churn pixels also pass, with old reads retained, one charged retiring generation, third-generation refusal/recovery and monitor-only registration excluded from production invalidation.
Preparation-device removal and adapter migration remain unqualified. Browser/other CPU producers and full installed delivery/latency/receiver/A/V/fleet qualification continue under #802/#517. Defaults and installed beta remain unchanged.
#517/#802 network preparation evidence, October 7
NDI real decoded CPU/GPU pixel parity and reconnect PASS. Actual SRT publisher restart and RTMP reconnect restore video/audio with fresh prepared epochs; recorded 1,580 and 1,060 frames respectively with zero recording missing/dropped frames. Independent source-pattern/audio checks pass. These functional tests do not prove per-frame acquisition/receiver delivery or A/V skew.
Frozen native tree ac4fc8652b0c3706b8ef276755e5c901eef6af9d, core SHA256 6BE022E7DA661C046A6DEFAF707C0508728C3723EA0CDA6193A511EED325EED7, tests SHA256 2C799556BB3539AEDDA7DAD56FBEB0BD46071E6D1C249AC6F6D94A2B32233D04.
Full native run: 1,497 PASS, one FAIL in StreamBackpressureRateProbe.NoDestinationSustainsThirtyAndSixtyFps (60 fps missed scheduled slots). Exact same frozen binary isolated repeat: one test PASS, no missed slots. Both runs retained; the full run is not relabelled clean. Attribution of the initial rate failure remains unconfirmed under #517. 18 portable evidence tests pass.
Original RTMP harness undefined-factory FAIL retained, prerequisite #817 corrected, actual repeated reconnect PASS. PR #818 remains draft pending CI/review; parents stay open. No beta installation, default enablement, latency/display/receiver or fleet qualification.
Browser CPU preparation merged in #816 at main 8022624, verified native tree a5424be613f95f460c13b237bf750769b9f23431 matches the tested candidate. All 1,497 native tests, 18 portable evidence tests and all 17 exact-head CI checks pass. The sanitizer build-tools stall was cancelled and retried successfully; no test gate was waived.
Actual opt-in WebView2 pixels match an independent CPU-upload compositor, with zero Program source uploads/scratch uploads/texture creation. Arrival before render polling, consumer recreation, blocked-reader polling and retained-reader removal pass. Real 120-second trials delivered 7,189 prepared and 7,192 control Program packets with zero buffer loss; one and four absorbed render overruns remain recorded. Sampled source age and host progress are not per-frame acquisition or end-to-end latency proof.
Network preparation continues in #818. Capacity/refusal reporting, device recovery and installed/per-frame/latency/A/V/fleet qualification remain open. Parents are not closed; defaults and installed beta remain unchanged.
Network CPU arrival preparation merged in #818, main 8c7a0bd, verified native tree ac4fc8652b0c3706b8ef276755e5c901eef6af9d matches the archived candidate. All 17 exact-head CI checks pass. Actual NDI decoded CPU/GPU pixel parity and reconnect pass. Actual RTMP reconnect and SRT publisher restart restore fresh prepared epochs and decoded audio, with independent source-pattern/audio recording checks and zero missing/dropped recording frames.
The original 1,497-pass/one-rate-probe-failure full run and its same-binary passing isolated repeat remain retained. Later capacity/initial-recovery superset candidates passed all 1,501 and 1,502 native tests. This is not per-frame transport acquisition/receiver/display or A/V skew qualification. The preexisting RTMP harness factory prerequisite #817 is merged, validated and closed.
Capacity and owner recovery continue in #819. Stalled network held-frame refresh and #517 final qualification remain open. Defaults and installed beta are unchanged; parents stay open.
PR #819 merged as bdf1de5 after all 17 final-head CI checks passed. Final tested head 815b0aa preserves native tree 8b0c5445c124590b9d65d4e236cc63a538e37410. All 1,503 native tests pass; focused real-texture capacity, initial setup and injected device-status recovery preserve original CPU bytes/identity and zero Program source uploads. Old read leases keep storage charged until drained. Physical device reset and adapter migration remain unqualified.
Core SHA256 1D715FF278B27320302F74D39312D9782C1B25FB26B339CFB58C84D6C5243F94; tests SHA256 0BE4AE622FE3B8AF187F40E15EF085EF9F53E7C4A0FF669DF4175E6323567DF2. Failed before runs are retained in local artifacts/cpu-device-owner-recovery-802-* and cpu-initial-consumer-recovery-802-*.
The independent OS-camera reference attempt is INVALID: Windows registration belongs to the installed October 5 beta, while the owned headless core was the development candidate. Core reports CAMERA_MACHINE_REGISTRATION_CONFLICT and receiver reports camera not enumerated. No continuity claim follows; registration/installed beta remain unchanged. Network stalled-held refresh, actual latency, display/receiver/A/V and installed/full-workload qualification remain open. Defaults remain off; neither #802 nor #517 is closed and no beta is cut.
PR #820 merged as a276770 after all 17 exact-head CI checks passed. Tested head 7e65d7e; native tree aa7b6cd822fcc20d4c40dd8f0c2d35fc933a0a1a. Both real local SRT/RTMP controls fail held-token replacement readiness without the worker; both repairs pass byte-exact CPU-reference pixels, original identity/observation time, zero Program source uploads, unchanged decode/audio counts and stale signal health, fresh decoder epochs after publisher restart, and removal without republishing. All 1,505 native tests pass. Exact binary hashes/source bundle and all failed setup/harness/stale-object trials are retained in artifacts/network-held-recovery-802-*.
Defaults and installed beta are unchanged; this does not qualify output continuity, latency, display/camera/A/V, physical device reset or fleet operation. Pending weak-entry reclamation is in PR #821 with FAIL-before/PASS-after and 1,506 combined native tests; final CI pending. #802/#517 remain open.
Final frozen-candidate comparisons and remaining latency evidence
#821 merged as 9c66633 after all17 exact-head CI checks passed. Original hosted fake-core handshake failure and targeted retry remain retained on #798. #822 OS-camera reference tooling also merged after17 exact-head checks; both real baseline/candidate-core camera references preserved all2,701 measured pixel identities and the original installed executable was restored by SHA.
The exact tested native tree89871076fad150f9416f2027125efbe7517ab54f / coreSHA5B8EB7AA62C9B906863E46F2F0BE2B49EF68772A45AC04597D46B103BD918527 completed24 reference trials: three inline/isolated pairs for each of mixed and single-large-capture, with preparation0/1. Frozen pre-rebase source b0d0865 is recorded in each manifest. Program1080p60, buffer2, GPUcapture0, eight fake1080p30 Zoom sources plus two real BGRA mappings1080p/1440p at60Hz, local Program recording. All preparation-enabled trials had zero buffer/recording losses. Mixed isolation reduced mean work from about1.97–2.02ms preparation0 to0.90–0.995ms preparation1; single capture from1.67–1.75ms to0.81–0.94ms. Absorbed render overruns remain separate.
One disabled-preparation/inline capture control failed with six buffer underruns and six recording missing identities; the other23 trials passed those boundaries. Raw losses occurred as three late-Preparing/expired-Ready head pairs; #825 retains the suspected wait-predicate amplification without blaming it for the initial GPU lateness. No failed trial was replaced.
Important latency qualification gap: sampled source-admission identities for the prepared screen capture lagged the requested CPU observation by p50 19.1819ms/p95 69.136ms in capture, and p50 20.5551ms/p95 55.5036ms in mixed. The two selected Zoom sources matched requested identities in all2,838 mixed samples each. This is selection-observation age, NOT actual acquisition-to-receiver latency or per-frame evidence. These values do not pass or waive the latency requirement. Full raw reports plus a retained comparison summary are in artifacts/cpu-source-final-821-*.
#823/#824 now implement the first bounded per-frame source request/draw/upload/completion and Program submission/readiness/delivery trace, with exact GPU-identity joining and loss-invalidated verdicts. Release build and7 native trace tests pass; full native suite and actual traced workloads are in progress. This remains draft and opt-in, with camera/display/A/V/acquisition/p95 overhead/installed/fleet evidence still missing. CPU preparation/isolation defaults and the installed beta remain unchanged. #802 and #517 stay open.
#823 / #517 qualification finding, retained without a PASS waiver
Matched same-binary timing experiment: source 879a287, native tree 33b8f0a96035f59ddbd4bfc314991855548e90e8, core SHA256 6C3715576855E064D3B056042ED415D774916D4227BF16DCC677371107BF7838. Timing collector enabled in both conditions; CPU preparation 1, GPU capture 0, buffer 2, eight fake 1080p30 I420 feeds, 1080p/1440p60 BGRA mappings, mixed Program at 1080p60, local recording; each inline/isolated trial measured 45 seconds after fixed 15-second warmup. Defaults/installed beta unchanged.
Trace-off trials passed output/recording/sample admission and timing-accounting checks. However the control already had substantial render tails: isolated mean 0.882 ms, p99 interval 11.749–12.730 ms, p99.9 interval 12.747–17.816 ms; two render completion overruns. Inline control had 64 overruns. These percentile intervals retain histogram/endpoint uncertainty and are not rounded point estimates.
Trace-on inline passed delivery/recording/raw trace. Trace-on isolated FAILED: five buffer underruns and five missing scheduled recording frames, four skipped CPU slots, nine render overruns; raw per-frame trace reports five misses/one gap with zero trace loss or export failures. Its high-percentile finite range was exceeded (>32.768 ms), so the timing distribution is INVALID rather than clamped to a healthy value. The matched cost comparison is INVALID because candidate delivery/recording failed; <1% trace overhead is not established.
The first divergence occurs 30.125–30.391 seconds into measurement. A 95.7085 ms CPU render-work interval precedes five misses for targets 2729–2733. Last timely Program submission 2728 occurred at 30.202053 seconds; next submission 2730 at 30.317059 seconds. Existing stage logs attribute this exceptional tick to Program 77.84 ms and monitor-request/build work 17.66 ms, not ingress (0.10 ms) or event drain (<0.114 ms maximum). The D3D renderer reports frame 2729: total 77.627 ms, encoder/virtual-camera/Program-buffer region 77.498 ms; drawing 0.040 ms, upload 0.009 ms, readback zero. Independently scheduled shell and multiview exports simultaneously reported keyed-mutex AcquireSync scopes around 77.022/77.004 ms despite zero requested timeout. This locates a GPU-handoff/API-stall region; it does NOT prove which producer call, driver scheduling, OS preemption or tracing caused it. Some other periodic tails are attributed to ingress fetch and require separate attribution.
No beta/default promotion or issue closure. Retain artifacts/render-work-823-first-{off,on}, raw captures, snapshots, stderr, exact manifest/source archive and artifacts/render-work-823-first-cost.json. Next focused diagnostic subdivision must distinguish producer mutex acquire, copy, release and remaining encoder-export work before changing the handoff. This evidence remains under the approved #823/#517 scope; it does not establish #825's separate wait-predicate amplification as the initiating cause.
Both default-off CPU attribution slices are merged and parent acceptance remains open: #830 at c7521ab and #831 at 0db5714. Each has 17 successful exact-head checks. Final #831 source f5987f2 passed all 1,520 native tests, 59 enabled runtime/state tests, and 24 portable checks; external network/NDI opt-ins were not executed. Frozen source/binary manifests and complete raw captures are retained locally.
Final clean normal-logging trials: 14,410 measured Program deliveries, zero buffer/recording/raw-trace losses. CPU-tail gates still fail: inline p99 9.792–10.065 ms; isolated 8.198–8.465 ms. p99.9 upper uncertainty bounds exceed 12.5 ms in both. The earlier normal-logging pair retained 34/27 render deadline misses. Post-window 7/5 Program misses remain in the complete final captures and are excluded only from the explicitly pinned measured-window verdict, not deleted.
290 retained slow publication records and 85 slow fetch waits were observed; 76 waits overlapped publication bodies for at least 95% of their duration in the same steady-clock domain. Large optional thumbnail encoding/JSON packaging still occurs under the runtime mutex shared by Program configured()/fetch. The CPU record set includes warmup/shutdown and both processes report 7 logger drops (0 truncated/sink failures), so absence is not complete negative evidence. This supports the reproducible serialization finding #832 and its focused off-lock packaging proposal; owner ranking is pending. It does not attribute the separate earlier 77 ms encoder stall/five losses or prove GPU duration/kernel wait owner.
The #830 merge inadvertently triggered GitHub issue closure from a negated closing-keyword phrase in its body. #823 has been reopened and both PR bodies corrected. Parent gates remain unmet: camera/shared/monitor/display trace coverage, calibrated content latency, <1% p95 trace cost, installed/real-meeting/A/V/resource/hardware qualification. No beta/default/settings/registration/installation change. Installed October 5 core hash reverified unchanged as A0F1E91D9D3041F01975CC6DE72A69DBEA7BEE8BC3916E961F58764771F7F5F0.
Additional finite qualification evidence from #835 / draft PR #837: exact same Release core (31a7648, SHA256 7395f2bb02db32049df58813822cd858699ffc8aa66361772ec5f583a7a61580), eight synthetic 1080p I420 Zoom sources, two 1080p/1440p BGRA mappings, mixed selected-source Program, 1080p60 Program/Preview/multiview and local MP4, CPU preparation enabled and two-frame Program buffer. Thirty measured seconds / five-second warmup per fresh core.
After correcting duplicate initial/end observations (#838), all sampled source-advancement judgments pass. Inline monitoring has 82 Program underruns / 82 missing recorded frames with grading demand zero, and two / two with three private grading monitors open. Isolated monitoring has zero delivery losses, recorded missing frames, queue drops and audio loss for both closed and open trials. Do not infer an improvement from these variable inline counts or attribute a closed-editor failure to active grading work. This does not qualify a release or identify the first divergent per-frame boundary.
The initial full native suite also retained one missed no-destination 1080p60 production slot (0.83 ms worst lateness); a fresh isolated repeat passed both 30 and 60 fps. Neither short repeats nor averages waive the initial failure. Detailed manifests, stderr, snapshots and recordings are retained locally; selected adapter, camera/stream receiver, physical display, real SDK and sustained/fleet evidence remain unverified.
No output buffering, monitor default, production quality/cadence or installed beta was changed.
Additional pinned synthetic delivery evidence from #835 / PR #837, not a closure or re-ranking of #517:
Two pairs of 60-second mixed-source 1080p60 Program/Preview/multiview + Program MP4 trials, each with ten seconds warmup, advanced source grades, CPU source preparation and a two-frame Program buffer. Isolated monitoring had zero measured Program/recording losses in both pairs with grading closed and both pairs with three grading editors/scopes open. The first inline trial in each setting lost two Program frames and two recorder frames; the second inline trial passed. Both failed controls remain failures. No encoder queue drops/audio lost samples were observed. Scope/source advancement is sampled, not per-frame display/receiver proof.
This comparison does not establish the cause of the inline losses, eliminate #517, or qualify final-head/sustained delivery. Original snapshots, stderr, MP4s and binary/harness/driver hashes are retained locally in artifacts/grading-advanced-{closed,open}-review; committed evidence describes their scope and limitations:
https://github.com/iamfatness/CoreVideoPro/blob/codex/source-grading-native-preview-835/docs/reference/source-grading-advanced-validation-2026-10-08.mdOctober 11 live requalification is now receiving all eight requested SDK video feeds (recording privilege was granted).
The installed-layout #802 candidate passed a three-minute cold-background smoke test: three newly opened 1920x1080 media assets reached admitted GPU images; the existing two-frame Program buffer had zero underruns, overflows, scheduled deadline misses, GPU-not-ready events and output identity gaps. Independent registered OS webcam pixels also passed: 11,101 measured samples, zero missing/repeated/reordered/torn identities, maximum arrival interval 20.584 ms. One compositor render overrun was absorbed without Program delivery loss.
The longer 30-minute run is still in progress. It has already failed combined zero-miss acceptance: the OS receiver recorded 37 repeated identities paired with 37 missing identities after the fixed 30-second warm-up. Program and publisher counters remain free of delivery losses/replaced pending frames/publication exceptions. Windows Frame Server reports 41 held frames total (40 contended shared-memory reads, one unchanged read), including four warm-up holds. This locates an independent downstream reader failure; it does not imply that #802 source preparation or the SDK crashed.
This repeats the camera-contention boundary already described in the parent render-delivery completion plan. The retry QA path must be qualified with actual races and matched off/on trials before enabling it. We are preserving the entire failed receiver capture and finishing the Program/cold-admission run. PR #862 remains draft. A successful receiver exit alone no longer produces a passing harness verdict: the maintained harness now drains and judges actual pixel evidence.
Completed October 11 requalification of the #802 candidate in the real test meeting: zero-miss acceptance failed. PR #862 remains draft.
Boundary Result SDK No crash/disconnect during 1800.034 measured seconds; eight feeds fresh at the end, all negotiated 1920x1080. Incoming source snapshot rejections remain separate from process stability. Cold admission All 20 new 1080p background identities reached completed admitted GPU images. Sampled Program texture uploads/creations/scratch all zero. Program 201 underruns, 132 overflows, 371 render deadline misses. Delivery-thread deadline/GPU-not-ready/sequence-gap counters stayed zero and do not override the underruns. OS webcam pixels 242 repeated, 173 missing, zero invalid/reordered identities; 108,302 samples over 1805.009485 measured seconds after fixed 30-second warm-up. Maximum interval 49.632 ms. Camera window includes five seconds after Program workload. Publisher Zero pending replacements/publication exceptions. Publication counters cannot prove receiver continuity. Monitor 181 pending jobs superseded after first measured sample; no worker failure. Display presentation unverified. Memory Minimum sampled commit headroom 80,820 MiB, limit 130,755 MiB; no recurrence of commit exhaustion. Two boundaries are demonstrated. Before any Program loss, camera read contention caused 37 repeats and 37 missing identities. Later Program losses began at measured second 1007.264; the first logged misses had an empty queue, with Preparing/Ready late cases later. This differs from the previously repaired expired-ready-head amplification. Slow-tick windows include 35.40 ms ingest and 18.96 ms monitor handoff, but do not establish which owner/work/preemption caused each loss. We do not yet have a complete causal attribution for these producer failures.
Next diagnosis: retain the exact workload/quality/two-frame buffer, enable the existing Zoom handoff wait/body timing and bounded delivery trace, correlate production submission/readiness with missed slots and monitor handoff. Reader repair still requires controlled off/on contention trials with its actual loaded DLL pinned. Do not flip retry defaults or add buffering to obtain a pass.
The maintained harness now judges receiver pixels before final acceptance. Seven pixel-judge tests pass, and the complete failed raw evidence is preserved under
artifacts/qa-802/live-installed-soak/on the reference machine. The sanitized qualification isqualification.json; the committed report records exact limits. All 17 CI checks passed on production revision eac0aae; the subsequent QA/documentation commit 20936ea is pushed and has a fresh CI run.- added a commit that references this issue
on Oct 11, 2026 PR #862 is merged to main at 491e3f7 under the owner's instruction to integrate all open PRs. All 17 CI/analysis checks passed on the combined revision 456d762, which included #858/#860; the merged main tree matches that tested revision exactly.
The retained 30-minute live qualification is still failed. Incremental source preparation/queue improvements are integrated; zero-miss delivery and the remaining producer/reader attribution remain open. No beta was published or running app replaced during these merges.
Evidence (live session 2026-09-13)
As the meeting grew to 16 participants (9 camera-on) with the Tiles wall at 1080p on Program, the monitor-shed engaged for the first time this session:
Program render cost hit 16.85 ms against the 16.67 ms one-frame budget. The shed protected Program by dropping monitor cadence to divisor 3 (as designed), and it recovered after ~5 min — but Program itself was at/over the ceiling.
Why it matters
This is the load ceiling for a full room with the wall at 1080p. The monitor-shed is a mitigation, not headroom: when Program alone approaches one frame, there is nothing left to shed. Rooms larger than ~16, or a screen-share on top of a full 1080p wall, will cross it.
Context
Ask
Profile the Program render at 16×1080p and find headroom (tile compositing cost, shared-texture reuse, per-tile shader work) before rooms scale further. Establish the real participant ceiling at 1080p on the reference GPU.
Found via the live-session metrics sweep 2026-09-13.