Editor video playback performance

[richiemcilroy]

This pull request contains changes generated by a Cursor Cloud Agent

 

Greptile Summary

Implemented a frame queue system with timing-based playback synchronization to improve video playback performance in the editor. The changes replace the previous single-frame rendering approach with a buffered queue that can hold up to 5 frames, intelligently dropping late frames (>40ms behind) and rendering frames at their target timestamps for smooth playback.

Key improvements:

• Replaced Rust watch channel with mpsc channel for better throughput (8-frame buffer)
• Added borrow() and readInto() methods to SharedArrayBuffer for zero-copy frame access
• Implemented frame timing metadata (frame_number and target_time_ns) throughout the pipeline
• Created dedicated stride correction worker to offload row padding work
• Increased SharedArrayBuffer from 4 slots × 8MB to 6 slots × 16MB
• Refactored render loop to use RAF with intelligent frame timing and dropping logic
• Optimized memory allocation by using Arc::clone in FFmpeg decoder instead of full data copies

Architecture changes:

• Moved stride padding removal from Rust to TypeScript (now handled in frame-worker or stride-correction-worker)
• Added resetFrameState() call on project config updates to reset playback timing
• Replaced Kobalte UI components with custom button implementations in ExportPage for simpler rendering

Confidence Score: 4/5

• Safe to merge with careful testing of edge cases around frame timing and buffer management
• The implementation is well-structured with good separation of concerns. The frame queue logic handles timing correctly, and the zero-copy optimizations are sound. However, the complexity of the timing synchronization logic and the introduction of worker-based stride correction adds surface area for potential race conditions or timing issues that should be validated through testing
• apps/desktop/src/utils/frame-worker.ts - complex frame timing logic and queue management should be tested thoroughly for edge cases

Important Files Changed

Filename	Overview
apps/desktop/src/utils/frame-worker.ts	Refactored from single-frame to queue-based rendering with frame timing synchronization for smooth playback
apps/desktop/src/utils/shared-frame-buffer.ts	Added readInto and borrow methods for zero-copy frame access, increased retry limit to prevent infinite loops
apps/desktop/src-tauri/src/frame_ws.rs	Added create_mpsc_frame_ws for mpsc channels, extended frame metadata with frame number and target time
apps/desktop/src-tauri/src/editor_window.rs	Replaced watch channel with mpsc channel, removed stride padding logic from Rust (now handled in TypeScript)
apps/desktop/src/utils/socket.ts	Added stride correction worker for direct canvas rendering, increased buffer size and slot count
crates/rendering/src/frame_pipeline.rs	Added frame number and target time metadata to RenderedFrame for playback synchronization

Sequence Diagram

sequenceDiagram
    participant Rust as Rust Rendering Pipeline
    participant WS as WebSocket Server
    participant SAB as SharedArrayBuffer
    participant FW as Frame Worker
    participant SCW as Stride Correction Worker
    participant Renderer as WebGPU/Canvas2D Renderer
    participant Canvas as Display Canvas

    Note over Rust: Decode & render frame
    Rust->>Rust: Add frame_number + target_time_ns
    Rust->>WS: Send frame via mpsc channel
    WS->>SAB: Write frame to shared buffer slot
    
    loop Poll Shared Buffer
        FW->>SAB: borrow() or readInto()
        SAB-->>FW: Frame bytes (zero-copy)
        FW->>FW: parseFrameMetadata()
        FW->>FW: Queue frame with timing info
    end
    
    loop Render Loop (RAF)
        FW->>FW: Calculate frame timing
        alt Frame too late (>40ms)
            FW->>FW: Drop frame from queue
        else Frame ready to render
            alt WebGPU path
                FW->>Renderer: renderFrameWebGPU()
                Renderer->>Canvas: Display frame
                FW->>SAB: release() borrowed frame
            else Canvas2D with stride != width*4
                FW->>SCW: Offload stride correction
                SCW-->>FW: Corrected frame data
                FW->>Renderer: putImageData()
                Renderer->>Canvas: Display frame
            else Canvas2D no correction needed
                FW->>Renderer: putImageData()
                Renderer->>Canvas: Display frame
            end
        end
    end

Loading

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[richiemcilroy]

@greptile review this