Skip to content

Mocap Slice B — ENABLE_MOCAP build infra + VideoFrameSource (Qt Multimedia: file / camera / image-sequence) #871

Description

@fernandotonon

Part of epic #869. Build infrastructure + the frame-source abstraction. No ML in this slice.

Goal

A new ENABLE_MOCAP build option that brings in Qt Multimedia, and a VideoFrameSource abstraction that delivers timestamped RGB frames from (a) a video file, (b) a live camera, (c) an image sequence (for headless tests) — so Slices C–F never touch Qt Multimedia directly.

CMake / build

  • New option ENABLE_MOCAP (default OFF) in CMakeLists.txt, near the ENABLE_ONNX/ENABLE_ALEMBIC blocks. It requires ENABLE_ONNX for the predictors (error out with a clear message if MOCAP is ON and ONNX is OFF) but VideoFrameSource itself only needs Qt Multimedia.
  • find_package(Qt6 COMPONENTS Multimedia) only when the flag is ON; link Qt6::Multimedia to the app + test targets. The QML camera preview (Slice F) will also need the QtMultimedia QML module at runtime — verify it deploys in the macOS bundle (macdeployqt picks it up when linked) and note the Linux/Windows deploy steps.
  • Guard every mocap source file with #ifdef ENABLE_MOCAP at the compilation-unit boundary the way AlembicImporter does (default build unaffected; src/CMakeLists.txt adds the src/Mocap/*.cpp files only under the flag).
  • CI: enable ENABLE_MOCAP on the Linux test/coverage lane only at first (the ENABLE_PS1_RIP precedent). Check whether the aqt Qt install in .github/workflows/deploy.yml needs the qtmultimedia module added (-m qtmultimedia or the archives list). Ubuntu runner needs the FFmpeg backend's runtime deps (Qt 6.5+ bundles FFmpeg in official binaries; verify under Xvfb).
  • Packaging notes (document, don't necessarily wire yet — Slice G finalizes): Debian control deps, macOS Info.plist.in gains NSCameraUsageDescription ("QtMeshEditor uses the camera for live performance capture."), Windows MinGW — verify Qt Multimedia MinGW binaries exist in the Qt version used; if not, MOCAP stays OFF on Windows initially (the ONNX/MinGW precedent).

src/Mocap/VideoFrameSource.{h,cpp}

One abstract interface + three implementations:

struct MocapFrame {
    QImage image;      // Format_RGB888, already converted
    double timeSec;    // media timestamp (file) or wall-clock since start (camera)
    qint64 frameIndex;
};

class VideoFrameSource : public QObject {
    Q_OBJECT
public:
    virtual bool open(QString* error) = 0;
    virtual void start() = 0;
    virtual void stop() = 0;
    virtual bool isLive() const = 0;
    virtual double nativeFps() const = 0;  // 0 if unknown
signals:
    void frameReady(const MocapFrame& frame);
    void finished();                        // file sources: end of media
    void errorOccurred(const QString& msg);
};
  1. FileFrameSourceQMediaPlayer + QVideoSink: connect QVideoSink::videoFrameChanged, map each QVideoFrame (toImage().convertToFormat(QImage::Format_RGB888)), carry QMediaPlayer::position() as the timestamp. Support a targetFps decimation option (skip frames closer than 1/targetFps to the last emitted one) — offline capture at 30 fps is plenty. Playback rate: leave real-time for v1 (a 60 s video takes 60 s) — document that faster-than-realtime decode is a known follow-up (QVideoSink is playback-driven).
  2. CameraFrameSourceQCamera + QMediaCaptureSession + QVideoSink. Constructor takes a QCameraDevice; expose a static availableDevices() returning id+description (feeds the GUI picker and the MCP list_capture_devices tool). Handle QCamera::errorOccurred and (macOS) permission denial → errorOccurred with a human-readable message. Latest-wins delivery: if the consumer is still busy when a new frame arrives, drop the old pending frame (an std::atomic-flagged single-slot mailbox), never queue — live inference must not fall behind.
  3. ImageSequenceFrameSource — takes a list of image paths + an fps; emits them synchronously on start(). This is the test double: every unit test and the CLI --frames-dir debug flag run through it, headless, no Qt Multimedia needed (compile it even without ENABLE_MOCAP… simpler: it lives with the others under the flag, and tests for it run in the mocap CI lane).

Also a small pure helper mocapFrameToRgbTensor(const QImage&, int targetW, int targetH, letterbox...) shared by both predictors — but the letterbox math itself belongs to Slice C's pure-data core; here just guarantee Format_RGB888 output.

Tests

  • VideoFrameSource_test.cpp: ImageSequenceFrameSource emits N frames with correct timestamps/order; decimation logic (feed 60 fps timestamps, target 30, assert ~half emitted); latest-wins mailbox drops intermediates (feed 3 frames while consumer blocked, assert only newest delivered).
  • File/camera sources: smoke-test open() failure paths (nonexistent file → error signal). Real camera tests are impossible in CI — guard with a QTMESH_MOCAP_CAMERA_TESTS env like the skin reference tests.

Acceptance criteria

  • cmake -DENABLE_MOCAP=ON -DENABLE_ONNX=ON builds on macOS + Linux CI; default build is bit-identical in behavior (no new deps pulled).
  • FileFrameSource plays an mp4 and delivers RGB888 frames with monotonically increasing timestamps (manual verification note in the PR).
  • ImageSequenceFrameSource + decimation + latest-wins covered by headless unit tests.
  • NSCameraUsageDescription added to Info.plist.in.
  • CI lane updated; deploy.yml Qt module list verified.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-assistLocal-AI-assisted 3D workflows (epic prefix: AI:)animationAnimation systems: skeletal, morph, pose, VAT, alembic, proceduralenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions