Skip to content

Document-mode live preview re-runs the film look's SVG filter on every video frame #11221

Open

Description

@atomantic

Problem

filmLookRuntimeSource() (server/lib/filmLook.js) installs the project's film look as one SVG <filter> on the document root of a composition page. That is right for the render (one evaluation per captured frame) and for a still, but the composition live preview (client/src/components/musicVideo/CompositionPreviewPlayer.jsx → the preview iframe, documentPreview.js) is a playing page: a browser re-runs an SVG filter chain over every decoded video frame, so a document-mode project with video takes and a saved look plays the same way the storyboard animatic did before #11220.

Measured in headless Chromium at phone width (390×219) on a 30 fps clip, filter on the box holding the video:

look video fps
none 30
Super 8 (15 primitives) 30
VHS (22 primitives) 3.6
Neon rain (49 primitives) 0.6

Adding the stages one at a time, the cost grows with the primitive count and the picture size (the full chain: 2.6 fps at half size, 0.2 fps at double size), not with any one stage. #11220 fixed the animatic (FilmLookFrame.jsx) by switching to filmLookMotionLayers() while the song plays; the document preview still takes the exact filter.

Decision

Give the runtime a playback mode that mirrors the animatic's: while the preview page is playing (the player's play/pause messages, portos-mv:*), the runtime removes the root filter and instead applies filmLookMotionLayers() — the CSS filter functions and the weave transform on the document root, and the overlay layers (cast washes, grain and tape tiles, vignette, leak) as fixed-position, pointer-events-none elements the runtime appends after the document's own layers, re-shifted from the same frame(n) hook. On pause and on every seek the runtime goes back to the exact filter, so a paused frame matches the render. The render path (documentRenderInitScripts → openComposition({ initScripts })) never plays, so it keeps the exact filter unchanged.

filmLookMotionLayers is not self-contained today (it uses module-level helpers), and the runtime inlines by toString(): either make it self-contained like filmLookFilterMarkup (the test in server/lib/filmLook.test.js that re-evaluates the function standalone is the pattern) or inline its helpers into the runtime source string.

Files

  • server/lib/filmLook.js — filmLookRuntimeSource, filmLookMotionLayers
  • client/src/components/musicVideo/CompositionPreviewPlayer.jsx — tell the runtime when the preview is playing (it already posts portos-mv:film-look)
  • server/services/musicVideo/documentPreview.js — where the preview page is assembled
  • docs/HTML_COMPOSITIONS.md — "Music-video film look" section

Acceptance

  • A document-mode project with video takes and the Neon rain look plays its live preview at the clip's frame rate in a headless-Chromium check like the one above (script pattern: a page with a looping <video>, requestVideoFrameCallback count over 3 s).
  • Paused or after a seek, the preview page's root carries the exact url(#portos-film-look) filter, and a document render's captured frames are byte-identical before and after the change (the render never enters playback mode).
  • The overlay elements the runtime adds never intercept pointer events and sit above every document layer, lyrics included.
  • filmLook.test.js covers the mode switch; the existing runtime tests still pass.

Model medium: routine multi-file work in a known pattern. Effort high: placing overlays correctly inside an arbitrary sandboxed document (stacking contexts, the lyric layer, the seek wrapper) needs care per step.

Activity

  1. added
    bugProposed from a runtime-safety audit
    planTracked by /do:replan
    in-progressClaimed and being worked
    on Oct 12, 2026
  2. self-assigned this
    on Oct 12, 2026
  3. atomantic commented on Oct 12, 2026

    @atomantic
    OwnerAuthor

    Current main now renders document previews in the OS-isolated worker and streams JPEG frames instead of mounting an iframe. The exact film filter still runs on every preview frame. I will carry playback mode with each queued preview seek so playing frames use the shared motion layers, while pause, scrub and frame steps restore the exact filter. Final render calls retain their existing exact mode; generated HTML remains isolated.

  4. atomantic commented on Oct 12, 2026

    @atomantic
    OwnerAuthor

    PR #11230's review follow-up fixed a reproduced one-frame seek snapback during playback and corrected a Windows-only path assumption in the renderer test. Focused validation passed (42 server/browser tests, 16 client tests), and local pregate passed.

    Throughput acceptance is still unresolved. A same-page headless-Chromium 390×219 synthetic 30 fps video probe counted requestVideoFrameCallback over 3 seconds, advancing the runtime film frame each callback: clean 16.20 fps (42 dropped), exact Neon rain 6.65 fps (67 dropped), motion Neon rain 15.27 fps (45 dropped), clean recheck 14.24 fps (44 dropped). The unfiltered controls did not reach clip rate, so these measurements cannot prove the requested 30 fps acceptance. They also do not measure the isolated preview worker's JPEG transport.

    Chosen follow-up: retain this issue open after merging the functional mode-switch improvement. Run the same clean/exact/motion control matrix in a provisioned environment where the clean 30 fps control sustains clip rate, then measure rendered JPEG frame delivery through the current worker/socket/client path (server/services/generatedPreview/sessions.js, client/src/hooks/useGeneratedPreview.js). If capture/transport limits delivery, optimize that existing isolated path without mounting generated HTML in the application browser. Acceptance remains a sustained clip-rate preview plus unchanged exact paused/render pixels and pointer-transparent overlay stacking. The PR now references this issue without closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:mediabugProposed from a runtime-safety auditeffort:highDispatch effort: highin-progressClaimed and being workedmodel:mediumMedium model neededplanTracked by /do:replan

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions