Repository navigation
feat(render): report encoding and assembling progress - #5137
Conversation
Edit accuracy: accurate 2059 (base branch 2059), smooth 1609 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (1)
|
…ess callback throws
…ded callback ending
7c5b46e to
22eecba
Compare
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at e059c680.
What I checked
- Stats parsing.
ffmpegStatsReaderkeeps the partial line between chunks and splits on both\rand\n, which is how ffmpeg writes its stats line. The encode and mux passes set no-logleveland no-nostats, so ffmpeg prints theframe=/time=stats on all three paths (disk, chunked, streamingclose()).runFfmpegandManagedChildProcessalready carriedonStderr, so the stderr plumbing needed no change. - Percent mapping. Disk encode goes from 75 to 90. The streaming and HDR paths encode from wherever capture left the bar (80), up to 90. Assemble goes from 90 to 99, and only
Render completereaches 100.updateJobStatusstill clamps to a monotonic max, sopctcannot go back. Assemble is a single pass on each branch (mux with audio, faststart without, or HLS), sodonenever resets inside the stage either. - Machine line. It is gated on
!isTTY, the Docker flag and a setstageProgress.--quiet(which includes batch JSON) has noonProgress, so no machine line can get into a JSON document. Docker passes the flag only when the host stdout is a TTY. - Telemetry codes. I stripped the counts from the labels in the producer. Two codes change shape: "Starting browsers (k/n ready)" and "Capturing/Streaming frame N/M (…)". Neither of those values was stable before this PR either.
Encoding framemaps toencoding_video, so the video-encode bucket is unchanged. - Reuse / simplicity. The repo has no existing ffmpeg progress parser and no existing machine-progress line. The studio server's SSE already reads
currentStage, so it picks up the richer labels without a change. The new pieces are small and each has real callers.
Tests
- Engine (runFfmpeg, streamingEncoder): 87 pass. CLI (render, progress, dockerRunArgs): 146 pass. Producer (shared, encodeStage, captureStreamingStage, assembleStage): 67 pass.
- Mutations: 10 of 12 were killed. They covered the split-line carry, the immediate count and the live count in
close(), the Docker TTY flag on both sides, removing counts from stage codes, the duplicate suppression, and the closing line on assemble, disk encode and streaming encode. - Two mutations survived:
- Dropping the
startNumber +chunk offset. This is already listed under "Not covered". - Removing the
Math.min(done, total)clamp inreportEncodeProgress.
- Dropping the
Nit (PR body only, the code is fine). The body says GIF and PNG-sequence failures now land in encoding_video. They do not. startEncodeProgress sets the label to Encoding GIF or Writing PNG sequence, and the frame-count report (encoded()) only runs after the encode succeeds. So a failed GIF or PNG encode still has that label at failure and still lands in encoding_gif or writing_png_sequence, as on main. That is arguably better than what the body describes. Worth correcting the sentence so nobody goes looking for a bucket move that never happened.
— Rames
What
hyperframes rendernow reports progress while ffmpeg encodes and assembles. Before, the bar sat at one number for the whole encode. On a 4K render that was 25 s of silence at 75%.Encode: the stage reads
Encoding frame N/Mand moves up to 90%: from 75% on the disk path, and from where capture left the bar on the streaming and HDR paths. The frame count is ffmpeg's ownframe=stat from stderr, for both the disk encoder (single and chunked) and the streaming encoder while it finishes after capture. GIF and PNG-sequence output report their start and their last frame.Assemble: the stage moves from 90% to 99% as the audio mux and MP4 faststart write output seconds (
time=). 100% still means the file is complete.Every stage closes: encode and assemble always end with a
done = totalline, even when ffmpeg's last count differs or a pass reports nothing (WebM, MOV, HLS).For programs: when stdout is not a terminal, every progress update also prints one machine line beside the unchanged human line:
codeis one ofcompile,extract_video,process_audio,start_browsers,capture,encode,assemble.done/totalare frames for capture and encode, seconds for assemble, and ready workers for start_browsers. They are left out when a stage has no count.pctnever goes back. Lines come at most 4 times a second, plus each stage change and stage end. This is the shape HyperFrames Desktop's export panel agreed to read. Terminal output is unchanged. The finished render is the exit code: there is no 100% machine line.--dockerin a terminal: output is unchanged. The container's stdout is a pipe even when the host's is a terminal, so the host passesHYPERFRAMES_STDOUT_IS_TTY=1and the container prints no machine lines.The producer carries the stage on
RenderJob.stageProgress, so the render API's progress callback gets it too. The render API already catches a callback that throws, so a bad callback cannot stop a render here either.Proof: a real 4K render
product-promofrom the registry,--resolution 4k, 600 frames, on Linux.Before (main): capture ended at 70%, then
75% Encoding videostayed unchanged for 25.5 s of encode, then the render completed.After (this branch): the same render, with timestamps from the piped log:
A second render with audio, the producer's
audio-mux-parityfixture at normal resolution, took the streaming path. It reported capture, then the encoder's last frames (246/300,300/300), then assemble from 0 to 10 s through the audio mux.Failure telemetry keeps its stage buckets. A failed render's
failed_stage_codeis built from the stage label at failure. Labels with live counts (Capturing frame 120/600 (6 workers), and nowEncoding frame 600/600) would give each render length its own code, so counts and parentheses are dropped before naming the stage. A video encode failure keeps its existing code,encoding_video. GIF and PNG-sequence encodes now report frames under the same label, so their failures also land inencoding_video(main hadencoding_gifandwriting_png_sequence); the event's output format still tells them apart. Capture failures, already unbounded on main, becomecapturing_frame,streaming_frameandlayered_composite_frame. The rawfailed_stagefield still carries the full label, as it did on main.Checks
close()waits; encode and assemble wiring and their closing lines (GIF, and a stage with no reporting pass); no progress for a zero total; the percent mapping; the machine line (piped vs terminal); the Docker terminal flag only for a terminal, and no machine line when it is set; the streaming stage's closing line; a PNG-sequence close. Each one fails when the code it covers is broken, and passes 3 runs in a row.Not covered
done/total.--batchwithout--jsonprints each row's lines in turn with no row id.