Editor video playback performance #1473

richiemcilroy · 2025-12-30T18:07:49Z

Summary

Refactored frame processing and shared buffer handling for better performance
Added frame timing metadata to frame pipeline
Optimized ImageData object reuse in socket utility
Added stride correction worker for image data processing
Improved WebGPU frame state reset and buffer borrowing
Refactored export UI and frame rendering logic

Originally merged to wrong branch in #1472.

Greptile Summary

Implements a comprehensive video playback performance overhaul replacing single-frame rendering with a queue-based system that buffers up to 5 frames with intelligent timing synchronization and late frame dropping (>40ms behind)
Moves stride correction (row padding removal) from Rust to TypeScript workers and optimizes memory usage by implementing zero-copy buffer access patterns, cached ImageData objects, and Arc-based reference counting in FFmpeg decoder
Adds frame timing metadata (frame_number, target_time_ns) throughout the pipeline from Rust rendering to frontend frame workers to enable precise frame synchronization and playback timing control

Important Files Changed

Filename	Overview
`apps/desktop/src/utils/frame-worker.ts`	Completely refactored frame rendering from single-frame to queue-based system with timing synchronization; critical for smooth video playback
`apps/desktop/src-tauri/src/editor_window.rs`	Replaced watch channels with MPSC channels and moved stride correction to frontend; affects core frame pipeline architecture
`apps/desktop/src/utils/socket.ts`	Major buffer size increase (4×8MB to 6×16MB) and ImageData caching optimization; potential memory impact needs review
`apps/desktop/src-tauri/src/frame_ws.rs`	Added MPSC WebSocket server and frame metadata; contains potential mutex deadlock in new implementation
`apps/desktop/src/utils/stride-correction-worker.ts`	New worker for stride correction moved from Rust; architectural shift requiring careful performance validation

Confidence score: 3/5

This PR introduces significant architectural changes to video frame processing that could impact performance and stability if not thoroughly tested
Score reflects complex timing synchronization logic, memory management changes (buffer size increases, zero-copy patterns), and architectural shifts (Rust to TypeScript processing) that require careful validation
Pay close attention to frame_ws.rs for the potential mutex deadlock issue and socket.ts for the substantial buffer size increase that could affect memory usage

Sequence Diagram

sequenceDiagram
    participant User
    participant Editor as Editor Component
    participant EditorInstance as EditorInstance
    participant Renderer as Renderer
    participant FrameWS as Frame WebSocket
    participant Worker as Frame Worker
    participant Decoder as Video Decoder
    
    User->>Editor: "Load project for editing"
    Editor->>EditorInstance: "create editor instance"
    EditorInstance->>Decoder: "spawn video decoders"
    EditorInstance->>Renderer: "initialize renderer"
    EditorInstance->>FrameWS: "create frame websocket"
    FrameWS->>Worker: "init shared buffer & canvas"
    
    User->>Editor: "seek to frame position"
    Editor->>EditorInstance: "emit render frame event"
    EditorInstance->>Decoder: "get frames for time"
    Decoder-->>EditorInstance: "return decoded frames"
    EditorInstance->>Renderer: "render frame with uniforms"
    Renderer->>Renderer: "composite layers (display, camera, cursor, effects)"
    Renderer-->>EditorInstance: "rendered frame data"
    EditorInstance->>FrameWS: "send frame via websocket"
    FrameWS->>Worker: "process frame with metadata"
    Worker->>Worker: "decode and queue frame"
    Worker-->>Editor: "frame rendered notification"
    
    User->>Editor: "start playback"
    Editor->>EditorInstance: "start playback with fps"
    EditorInstance->>EditorInstance: "spawn playback loop"
    loop Playback Loop
        EditorInstance->>Decoder: "get frames for current time"
        EditorInstance->>Renderer: "render frame"
        EditorInstance->>FrameWS: "send frame"
        FrameWS->>Worker: "process frame"
        Worker-->>Editor: "display frame"
        EditorInstance->>Editor: "update playhead position"
    end
    
    User->>Editor: "export video"
    Editor->>EditorInstance: "render video to channel"
    loop Export Loop
        EditorInstance->>Decoder: "get segment frames"
        EditorInstance->>Renderer: "render with export settings"
        Renderer-->>EditorInstance: "rendered frame"
        EditorInstance->>Editor: "export progress update"
    end

Context used:

Context from dashboard - CLAUDE.md (source)
Context from dashboard - AGENTS.md (source)

Remove frame padding stripping logic. Update frame processing to use stride and raw data. Improve shared buffer consumer with readInto and slot size. Increase shared buffer slot size. Update WebGPU renderer to use stride for bytesPerRow. Use Arc for decoded frame data to avoid cloning. Co-authored-by: richiemcilroy1 <richiemcilroy1@gmail.com>

…k-performance-6d4c Editor video playback performance

greptile-apps

Additional Comments (1)

apps/desktop/src-tauri/src/frame_ws.rs, line 58 (link)

style: Inconsistent logging - both println! and tracing::info! are used for the same message

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{14 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2025-12-30T18:14:46Z

apps/desktop/src/utils/stride-correction-worker.ts

+	const result = correctionBuffer.slice(0, expectedLength);
+	const response: StrideCorrectionResponse = {
+		type: "corrected",
+		buffer: result.buffer,
+		width,
+		height,
+	};
+	self.postMessage(response, { transfer: [result.buffer] });


style: The .slice() operation creates a full copy of the corrected data, negating the performance benefits of buffer reuse. Consider using correctionBuffer.subarray(0, expectedLength).slice() or transfer the original buffer directly if the worker doesn't need it anymore.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop/src/utils/stride-correction-worker.ts Line: 46:53 Comment: **style:** The `.slice()` operation creates a full copy of the corrected data, negating the performance benefits of buffer reuse. Consider using `correctionBuffer.subarray(0, expectedLength).slice()` or transfer the original buffer directly if the worker doesn't need it anymore. <sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub> How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-30T18:14:52Z

apps/desktop/src-tauri/src/frame_ws.rs

+            let frame_opt = {
+                let mut rx = frame_rx.lock().await;
+                tokio::select! {
+                    biased;
+                    msg = socket.recv() => {
+                        match msg {
+                            Some(Ok(Message::Close(_))) | None => {
+                                tracing::info!("WebSocket closed");
+                                break;
+                            }
+                            Some(Ok(_)) => {
+                                continue;
+                            }
+                            Some(Err(e)) => {
+                                tracing::error!("WebSocket error: {:?}", e);
+                                break;
+                            }
+                        }
+                    },
+                    frame = rx.recv() => frame,
+                }
+            };


logic: Potential deadlock: holding the mutex lock across the entire tokio::select! block could block other tasks trying to access the frame receiver. Is this intentional to ensure atomic processing of each frame, or could the lock scope be reduced?

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop/src-tauri/src/frame_ws.rs Line: 178:199 Comment: **logic:** Potential deadlock: holding the mutex lock across the entire `tokio::select!` block could block other tasks trying to access the frame receiver. Is this intentional to ensure atomic processing of each frame, or could the lock scope be reduced? How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-30T18:14:53Z

apps/desktop/src/utils/frame-worker.ts

+	for (let i = 0; i < frameQueue.length; i++) {
+		const frame = frameQueue[i];
+		const frameNum = frame.timing.frameNumber;
+
+		const isSeek =
+			playbackStartTime !== null &&
+			lastRenderedFrameNumber >= 0 &&
+			(frameNum < lastRenderedFrameNumber ||
+				frameNum > lastRenderedFrameNumber + 30);


style: The seek detection logic uses a hardcoded threshold of 30 frames which may not work correctly for all frame rates. Should the seek detection threshold be based on frame rate or time duration instead of a fixed frame count?

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop/src/utils/frame-worker.ts Line: 436:444 Comment: **style:** The seek detection logic uses a hardcoded threshold of 30 frames which may not work correctly for all frame rates. Should the seek detection threshold be based on frame rate or time duration instead of a fixed frame count? <sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub> How can I resolve this? If you propose a fix, please make it concise.

greptile-apps · 2025-12-30T18:14:53Z

apps/desktop/src/utils/frame-worker.ts

+				fallbackIndex = i;
+			}
+
+			if (diffMs < -LATE_THRESHOLD_MS && !isSeek) {


logic: Frame dropping threshold of 40ms may be too aggressive for lower frame rates where frames naturally arrive >40ms apart. Should LATE_THRESHOLD_MS be adjusted based on the target frame rate to avoid dropping frames that aren't actually late?

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop/src/utils/frame-worker.ts Line: 466:466 Comment: **logic:** Frame dropping threshold of 40ms may be too aggressive for lower frame rates where frames naturally arrive >40ms apart. Should LATE_THRESHOLD_MS be adjusted based on the target frame rate to avoid dropping frames that aren't actually late? How can I resolve this? If you propose a fix, please make it concise.

cursoragent and others added 14 commits December 30, 2025 03:23

Merge branch 'main' into cursor/editor-video-playback-performance-6d4c

172272e

Fix type casting for buffer sources in rendering utils

0e60c96

Add frame timing metadata to frame pipeline and WS

5f70bf8

Add frame state reset and buffer borrowing for WebGPU

b40d0be

Add stride correction worker for image data processing

6a2bc34

Optimize ImageData object reuse in socket utility

8b025aa

Start render loop when queuing new frames

8403b50

Increase pollSharedBuffer timeout delay to 1ms

ab62608

Limit retries in advanceReadIndexCAS loop

ba81a76

Refactor frame handling to use a frame queue

3a28f0a

Refactor export UI and improve frame rendering logic

f1f0509

fmt

2435f66

Merge pull request #1472 from CapSoftware/cursor/editor-video-playbac…

07e770a

…k-performance-6d4c Editor video playback performance

richiemcilroy merged commit 472847c into main Dec 30, 2025
16 of 17 checks passed

greptile-apps bot reviewed Dec 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Editor video playback performance #1473

Editor video playback performance #1473

Uh oh!

richiemcilroy commented Dec 30, 2025 •

edited by greptile-apps bot

Loading

Uh oh!

Uh oh!

greptile-apps bot left a comment •

edited

Loading

Uh oh!

greptile-apps bot Dec 30, 2025

Uh oh!

greptile-apps bot Dec 30, 2025

Uh oh!

greptile-apps bot Dec 30, 2025

Uh oh!

greptile-apps bot Dec 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Editor video playback performance #1473

Editor video playback performance #1473

Uh oh!

Conversation

richiemcilroy commented Dec 30, 2025 • edited by greptile-apps bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Greptile Summary

Important Files Changed

Confidence score: 3/5

Sequence Diagram

Uh oh!

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (1)

Uh oh!

greptile-apps bot Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richiemcilroy commented Dec 30, 2025 •

edited by greptile-apps bot

Loading

greptile-apps bot left a comment •

edited

Loading