perf(studio): preview decodes hard-to-play videos at the size they are shown

[miguel-heygen]

What

Studio previews a video it cannot play directly (HEVC, ProRes) from a copy sized to how large the video shows on screen, instead of a copy at the source's full size. A 4K portrait clip shown in a normal Studio window is now decoded at about 512x912 instead of 2160x3840. Renders, hyperframes play, the static server and publish still use the source-size copy.

Why

Back-to-back 4K HEVC clips flashed the composition background at their cuts in Studio. The browser has to start decoding the next clip right at the cut, and with 4K copies it often had no frame ready in time. In matched runs, the same fixture with 720p copies never flashed, and with 4K copies it flashed at every cut. Decoding pixels nobody sees was the cost.

Related work

None. The gapless-cut frame preparation work is separate and not needed for this fix: the runs below use main's runtime.

How

• Player reports how large the composition frame shows on screen (its fit scale, plus any zoom the page puts around it) to the runtime: when it rescales, when the runtime is ready, and on play.
• Runtime waits for that number before asking for a copy, then asks for ?hf-proxy-box=WxH: the video's box (at least the stage) times the on-screen scale times devicePixelRatio, rounded up to a ladder of sizes (256 x 2^(k/2)) so nearby sizes share one cached copy. If the video later needs more pixels (fullscreen, a larger player, zoom then play), it moves to a larger copy; it never moves to a smaller one. A page shown on its own needs no host; a host that never reports gets today's source-size copy after one second.
• Studio route accepts only ladder sizes, passes the box to the shared proxy owner, and salts the ETag with it.
• Proxy owner (resolveProxy) scales the copy so its tighter side fills the box, never above the source, and keeps the box in the cache key. Without a box nothing changes.
• Studio no longer pre-warms source-size copies when it serves the page: the page asks for the right size once its layout is known. The first frame of a cold project waits for that copy. CLI surfaces keep their pre-warm.

Studio also reports the scale when a preview zoom settles (a transform the player cannot see) and for composition hover previews, which have no player.

Known limits:

• A box at or above the source makes its own source-size copy (byte-identical to the unbounded one).
• hyperframes play ignores the size, but each resize that crosses a larger size step reloads its clips.
• Moving the window to a screen with a different pixel density is picked up on the next resize or play.
• Copies never shrink: after zooming in, returning to Fit keeps the larger copies for the session.
• A zoom that settles while a reload preview is loading reaches it on the next play.
• During a resize drag the server still makes the intermediate sizes it was asked for; only the newest is loaded.

Test plan

Matched runs in Studio on a fixture of fourteen 4K HEVC clips cut back to back (built from the gapless-video-cuts recipe, magenta root so a missing picture shows), headless Chrome 152 at 1200x900, same machine, same fixture, main and this branch alternating, after one warm-up each. A frame counts when a reference square driven by the transport says playback is between the first and last cut.

	Background frames at cuts, per run	Median	Play enabled after open (median)
main (2160x3840 copies)	11, 21, 35, 19, 18	19	2.41 s
this branch (512x912 copies)	0, 0, 0, 0, 0	0	0.93 s

All 13 cuts observed in every run, no wrong-clip frames, no page errors. The server's cache held fourteen 512x912 copies for the branch and fourteen 2160x3840 for main.

Same setup on a heavily loaded machine (load average 20 to 26 from other jobs), all three builds interleaved, 5 runs each: main 127, 54, 61, 35, 76 (median 61); first commit 2, 0, 0, 0, 9 (median 0); final head 23, 0, 0, 0, 2 (median 0; the 23 run witnessed 12 of 13 cuts). Under load the sized copies still miss a cut now and then, about 30 times less often than main.

Cold start (empty cache, 3 runs each): Play enabled at 5.8 to 8.2 s on main and 2.9 to 4.2 s here; neither is clean during that first play, because copies are still being made (179 to 182 background frames on main, 19 to 74 here).

• Unit tests: ladder and parsing; runtime waits for the host's scale, sizes the request, falls back after 1 s, upgrades and never shrinks, loads only the newest size, regrows a first copy; player reports the scale on fit and play; Studio reports it on zoom settle and for hover previews; route validates, passes and tags the size; Studio does not pre-warm; a real-ffmpeg test checks the copy's dimensions. Changed files 3 runs green.
• Each new test fails with its line of the fix removed (11 mutants). The producer ffprobe argv contract and the comment ratchet pass.
• Manual: real Studio runs above.
• Comments follow CONTRIBUTING.md.

Before

main: mid-cut, the composition background (magenta in the fixture) shows instead of the incoming clip, and the timeline filmstrip is still empty.

main-3.mp4

After

This branch, same moment of the same run setup: the clip shows and the filmstrip is filled.

branch-3.mp4

[github-actions]

Edit accuracy: accurate 2059 (base branch 2059), smooth 1660 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[jrusso1020]

Approving at d78cabe0. All required checks pass in the latest run at this head.

What the diff does: Studio's preview proxies for hard-to-decode sources (HEVC/ProRes) are now sized to the on-screen box instead of the source size. Fewer background frames at a cut follow from cheaper decodes. The code doesn't hold or pre-roll frames, and the body is upfront that the effect is probabilistic. No path shows a stale or wrong-clip frame; an unready frame still falls back to the composition background, as before.

Body claims I checked against source:

• Studio passes prewarm=false (studio-server/src/helpers/mediaProxyPreview.ts:143), while the CLI's re-export keeps the default prewarm=true.
• hyperframes play calls resolveProxy without a box (cli/src/commands/play.ts:269).
• The route only accepts ladder sizes, and returns 404 for an off-ladder size or a box without hf-proxy before any transcode.
• With no box, the cache key and ETag salt are byte-identical to before, so existing caches stay valid.
• Render output can't change: render mode returns early, and publish rewrites to the baked _proxy/ files.
• The new parameters on resolveProxy, proxyEtagSalt and injectMediaCodecMapIntoHtml are optional with the old defaults, so existing callers are unaffected (checkBrowser.ts, staticProjectServer.ts, play.ts, publishProxyBake.ts).

Non-blocking:

1. An upgrade to a larger copy reloads the clip on screen. keepSized → loadProxy sets src and calls load() (core/src/runtime/mediaProxy.ts:109-146), which empties the element until the new file loads and is seeked. So on a panel/window resize, fullscreen, a zoom-in settling (NLEPreview.tsx:162), or play() after the layout grew (hyperframes-player.ts:434), every boxed clip reloads at once and can show the composition background for a moment. That's the symptom this PR targets. The upgrade path also skips registerSeekCompletion, which the first swap uses (line 350). Better: load the larger copy into a detached <video> and swap after it seeks, or defer upgrades for visible/playing clips. At minimum, register seek completion on upgrade.
2. shownBox measures once at swap time (mediaProxy.ts:92), so a clip that later scales up via its own transform keeps a copy sized for its starting box.
3. boxedElements is a strong Map (mediaProxy.ts:76), pruned only in setProxyDisplayScale, so detached videos stay referenced until the next scale report.
4. Scope: the diff also adds a public player→runtime message (set-display-scale) and new @hyperframes/core exports (PREVIEW_PROXY_BOX_PARAM, formatPreviewProxyBox, parsePreviewProxyBox). The studio scope in the title understates that.

CI:

• Timeline viewport gate (not required): it failed the first run on interaction p95 (59.3 / 58.3 ms) and passed the re-run at exactly 58.3. The same failure shows up in 5 of the last 17 gated main runs and in unrelated PRs, so it's a known flake.
• Tests on windows-latest: the first run failed the engine-cli lane on npxCommand.test.ts with spawnSync cmd.exe ETIMEDOUT. That's a known cold-start flake this PR doesn't touch, and the re-run passed. The navigation timeout (gsapValueAtPlayhead.browser.test.ts) came from the previous head, not this one.

— Rames