Add synthetic recording test framework and improve frame converter reliability

[richiemcilroy]

introduces a comprehensive synthetic testing framework for the recording pipeline, allowing CI testing without physical hardware. Also includes several reliability and thread-safety improvements to the frame converter infrastructure.

Greptile Summary

This PR introduces a comprehensive synthetic testing framework for the recording pipeline and significantly improves frame converter reliability through thread-safety enhancements and performance optimizations.

Key Changes:

• Synthetic Test Framework: New test_sources module provides synthetic video and audio generation with multiple test patterns (SMPTE bars, color gradients, frame counters, waveforms), validation framework to verify recording quality, A/V sync checking, and fragment integrity verification
• Frame Converter Thread-Safety: SwscaleConverter now uses thread_local contexts to avoid data races when used from multiple threads, VideoToolboxConverter protects session access with Mutex, all converters implement convert_into() for buffer reuse
• Frame Pooling: New VideoFramePool reduces allocation overhead, AsyncConverterPool integrates pooling to reuse output buffers, significant performance improvement for high-throughput scenarios
• Encoder Improvements: Hardware encoder throughput estimation prevents overload at high resolutions/fps, new HighThroughput preset forces software encoder when needed, proper color metadata (BT.709) now set on encoded streams, frame pooling in H264 encoder for conversion buffer reuse
• Testing Infrastructure: CLI test runner with multiple test suites (quick, full, resolution-specific), integration tests validate the complete recording pipeline, enables CI testing without physical hardware

Impact:

This enables automated testing of the recording pipeline in CI environments and resolves thread-safety issues that could cause data races in multi-threaded converter usage. The frame pooling significantly reduces memory allocation overhead in high-throughput scenarios.

Confidence Score: 4/5

• This PR is safe to merge with good testing coverage for the new synthetic framework and solid thread-safety improvements
• The PR introduces substantial new functionality with comprehensive testing infrastructure. The thread-safety improvements are well-designed using thread_local and Mutex appropriately. Frame pooling is a solid optimization. The encoder selection logic with hardware capability estimation is thoughtful. One minor concern: the SendableContext wrapper uses unsafe code with UnsafeCell but is justified because ThreadLocal ensures each thread gets its own context. The validation framework is thorough. Overall high quality work that significantly enhances testability and reliability.
• Pay attention to crates/frame-converter/src/swscale.rs (unsafe code with UnsafeCell), crates/enc-ffmpeg/src/video/h264.rs (encoder selection heuristics), and validation logic in crates/recording/src/test_sources/validation.rs

Important Files Changed

Filename	Overview
crates/frame-converter/src/swscale.rs	thread-local contexts for thread-safety, proper unsafe implementations
crates/frame-converter/src/videotoolbox.rs	refactored frame copying to accept output buffer, improved performance with reusable buffers
crates/frame-converter/src/pool.rs	added frame pooling support to async converter pool, improved resource management
crates/enc-ffmpeg/src/video/h264.rs	improved encoder selection with throughput estimation, added HighThroughput preset, better color metadata, frame pooling for conversions
crates/recording/src/test_sources/validation.rs	comprehensive validation framework for synthetic recordings with A/V sync checks
crates/recording/examples/synthetic-test-runner.rs	CLI tool to run synthetic recording tests with various configurations

Sequence Diagram

sequenceDiagram
    participant Test as Test Runner
    participant Video as TestPatternVideoSource
    participant Audio as SyntheticAudioSource
    participant Pipeline as OutputPipeline
    participant Pool as AsyncConverterPool
    participant Converter as FrameConverter
    participant Encoder as H264Encoder
    participant Validator as RecordingValidator

    Test->>Video: setup(config, video_tx)
    Video->>Video: spawn video generator task
    Test->>Audio: setup(config, audio_tx)
    Audio->>Audio: spawn audio generator task
    
    Test->>Pipeline: start recording
    
    loop Frame Generation
        Video->>Video: generate_video_frame(pattern, frame_number)
        Video->>Pipeline: send(FFmpegVideoFrame)
        Audio->>Audio: generate_audio_samples(generator)
        Audio->>Pipeline: send(AudioFrame)
    end
    
    Pipeline->>Pool: submit(frame, sequence)
    Pool->>Pool: get frame from pool (if enabled)
    Pool->>Converter: convert_into(input, pooled_output)
    
    alt Thread-Safe Conversion
        Converter->>Converter: SwscaleConverter: get_or_create thread_local context
        Converter->>Converter: VideoToolboxConverter: lock session, convert
        Converter->>Converter: D3D11Converter: use dedicated device
    end
    
    Pool->>Pipeline: return ConvertedFrame
    Pipeline->>Encoder: queue_frame(converted)
    
    alt Hardware Selection
        Encoder->>Encoder: estimate_hw_encoder_max_fps()
        Encoder->>Encoder: requires_software_encoder()
        alt High Throughput Needed
            Encoder->>Encoder: use software encoder (libx264)
        else Hardware Can Keep Up
            Encoder->>Encoder: use hardware encoder (videotoolbox/nvenc)
        end
    end
    
    Encoder->>Encoder: reuse converted_frame_pool
    Encoder->>Encoder: encode with color metadata
    Encoder->>Pipeline: write encoded packet
    
    Test->>Pipeline: stop recording
    Pipeline->>Video: stop()
    Pipeline->>Audio: stop()
    Pipeline->>Encoder: flush()
    
    Test->>Validator: validate(output_path, test_config)
    Validator->>Validator: probe video/audio streams
    Validator->>Validator: check frame count, duration
    Validator->>Validator: verify A/V sync offset
    Validator->>Validator: validate fragment integrity
    Validator->>Test: return ValidationResult

Loading