Part of epic #869. Build infrastructure + the frame-source abstraction. No ML in this slice.
Goal
A new ENABLE_MOCAP build option that brings in Qt Multimedia, and a VideoFrameSource abstraction that delivers timestamped RGB frames from (a) a video file, (b) a live camera, (c) an image sequence (for headless tests) — so Slices C–F never touch Qt Multimedia directly.
CMake / build
- New option
ENABLE_MOCAP (default OFF) in CMakeLists.txt, near the ENABLE_ONNX/ENABLE_ALEMBIC blocks. It requires ENABLE_ONNX for the predictors (error out with a clear message if MOCAP is ON and ONNX is OFF) but VideoFrameSource itself only needs Qt Multimedia.
find_package(Qt6 COMPONENTS Multimedia) only when the flag is ON; link Qt6::Multimedia to the app + test targets. The QML camera preview (Slice F) will also need the QtMultimedia QML module at runtime — verify it deploys in the macOS bundle (macdeployqt picks it up when linked) and note the Linux/Windows deploy steps.
- Guard every mocap source file with
#ifdef ENABLE_MOCAP at the compilation-unit boundary the way AlembicImporter does (default build unaffected; src/CMakeLists.txt adds the src/Mocap/*.cpp files only under the flag).
- CI: enable
ENABLE_MOCAP on the Linux test/coverage lane only at first (the ENABLE_PS1_RIP precedent). Check whether the aqt Qt install in .github/workflows/deploy.yml needs the qtmultimedia module added (-m qtmultimedia or the archives list). Ubuntu runner needs the FFmpeg backend's runtime deps (Qt 6.5+ bundles FFmpeg in official binaries; verify under Xvfb).
- Packaging notes (document, don't necessarily wire yet — Slice G finalizes): Debian control deps, macOS
Info.plist.in gains NSCameraUsageDescription ("QtMeshEditor uses the camera for live performance capture."), Windows MinGW — verify Qt Multimedia MinGW binaries exist in the Qt version used; if not, MOCAP stays OFF on Windows initially (the ONNX/MinGW precedent).
src/Mocap/VideoFrameSource.{h,cpp}
One abstract interface + three implementations:
struct MocapFrame {
QImage image; // Format_RGB888, already converted
double timeSec; // media timestamp (file) or wall-clock since start (camera)
qint64 frameIndex;
};
class VideoFrameSource : public QObject {
Q_OBJECT
public:
virtual bool open(QString* error) = 0;
virtual void start() = 0;
virtual void stop() = 0;
virtual bool isLive() const = 0;
virtual double nativeFps() const = 0; // 0 if unknown
signals:
void frameReady(const MocapFrame& frame);
void finished(); // file sources: end of media
void errorOccurred(const QString& msg);
};
FileFrameSource — QMediaPlayer + QVideoSink: connect QVideoSink::videoFrameChanged, map each QVideoFrame (toImage().convertToFormat(QImage::Format_RGB888)), carry QMediaPlayer::position() as the timestamp. Support a targetFps decimation option (skip frames closer than 1/targetFps to the last emitted one) — offline capture at 30 fps is plenty. Playback rate: leave real-time for v1 (a 60 s video takes 60 s) — document that faster-than-realtime decode is a known follow-up (QVideoSink is playback-driven).
CameraFrameSource — QCamera + QMediaCaptureSession + QVideoSink. Constructor takes a QCameraDevice; expose a static availableDevices() returning id+description (feeds the GUI picker and the MCP list_capture_devices tool). Handle QCamera::errorOccurred and (macOS) permission denial → errorOccurred with a human-readable message. Latest-wins delivery: if the consumer is still busy when a new frame arrives, drop the old pending frame (an std::atomic-flagged single-slot mailbox), never queue — live inference must not fall behind.
ImageSequenceFrameSource — takes a list of image paths + an fps; emits them synchronously on start(). This is the test double: every unit test and the CLI --frames-dir debug flag run through it, headless, no Qt Multimedia needed (compile it even without ENABLE_MOCAP… simpler: it lives with the others under the flag, and tests for it run in the mocap CI lane).
Also a small pure helper mocapFrameToRgbTensor(const QImage&, int targetW, int targetH, letterbox...) shared by both predictors — but the letterbox math itself belongs to Slice C's pure-data core; here just guarantee Format_RGB888 output.
Tests
VideoFrameSource_test.cpp: ImageSequenceFrameSource emits N frames with correct timestamps/order; decimation logic (feed 60 fps timestamps, target 30, assert ~half emitted); latest-wins mailbox drops intermediates (feed 3 frames while consumer blocked, assert only newest delivered).
- File/camera sources: smoke-test
open() failure paths (nonexistent file → error signal). Real camera tests are impossible in CI — guard with a QTMESH_MOCAP_CAMERA_TESTS env like the skin reference tests.
Acceptance criteria
Part of epic #869. Build infrastructure + the frame-source abstraction. No ML in this slice.
Goal
A new
ENABLE_MOCAPbuild option that brings in Qt Multimedia, and aVideoFrameSourceabstraction that delivers timestamped RGB frames from (a) a video file, (b) a live camera, (c) an image sequence (for headless tests) — so Slices C–F never touch Qt Multimedia directly.CMake / build
ENABLE_MOCAP(default OFF) inCMakeLists.txt, near theENABLE_ONNX/ENABLE_ALEMBICblocks. It requiresENABLE_ONNXfor the predictors (error out with a clear message if MOCAP is ON and ONNX is OFF) butVideoFrameSourceitself only needs Qt Multimedia.find_package(Qt6 COMPONENTS Multimedia)only when the flag is ON; linkQt6::Multimediato the app + test targets. The QML camera preview (Slice F) will also need theQtMultimediaQML module at runtime — verify it deploys in the macOS bundle (macdeployqtpicks it up when linked) and note the Linux/Windows deploy steps.#ifdef ENABLE_MOCAPat the compilation-unit boundary the wayAlembicImporterdoes (default build unaffected;src/CMakeLists.txtadds thesrc/Mocap/*.cppfiles only under the flag).ENABLE_MOCAPon the Linux test/coverage lane only at first (theENABLE_PS1_RIPprecedent). Check whether theaqtQt install in.github/workflows/deploy.ymlneeds theqtmultimediamodule added (-m qtmultimediaor the archives list). Ubuntu runner needs the FFmpeg backend's runtime deps (Qt 6.5+ bundles FFmpeg in official binaries; verify under Xvfb).Info.plist.ingainsNSCameraUsageDescription("QtMeshEditor uses the camera for live performance capture."), Windows MinGW — verify Qt Multimedia MinGW binaries exist in the Qt version used; if not, MOCAP stays OFF on Windows initially (the ONNX/MinGW precedent).src/Mocap/VideoFrameSource.{h,cpp}One abstract interface + three implementations:
FileFrameSource—QMediaPlayer+QVideoSink: connectQVideoSink::videoFrameChanged, map eachQVideoFrame(toImage().convertToFormat(QImage::Format_RGB888)), carryQMediaPlayer::position()as the timestamp. Support atargetFpsdecimation option (skip frames closer than1/targetFpsto the last emitted one) — offline capture at 30 fps is plenty. Playback rate: leave real-time for v1 (a 60 s video takes 60 s) — document that faster-than-realtime decode is a known follow-up (QVideoSink is playback-driven).CameraFrameSource—QCamera+QMediaCaptureSession+QVideoSink. Constructor takes aQCameraDevice; expose a staticavailableDevices()returning id+description (feeds the GUI picker and the MCPlist_capture_devicestool). HandleQCamera::errorOccurredand (macOS) permission denial →errorOccurredwith a human-readable message. Latest-wins delivery: if the consumer is still busy when a new frame arrives, drop the old pending frame (anstd::atomic-flagged single-slot mailbox), never queue — live inference must not fall behind.ImageSequenceFrameSource— takes a list of image paths + an fps; emits them synchronously onstart(). This is the test double: every unit test and the CLI--frames-dirdebug flag run through it, headless, no Qt Multimedia needed (compile it even withoutENABLE_MOCAP… simpler: it lives with the others under the flag, and tests for it run in the mocap CI lane).Also a small pure helper
mocapFrameToRgbTensor(const QImage&, int targetW, int targetH, letterbox...)shared by both predictors — but the letterbox math itself belongs to Slice C's pure-data core; here just guaranteeFormat_RGB888output.Tests
VideoFrameSource_test.cpp:ImageSequenceFrameSourceemits N frames with correct timestamps/order; decimation logic (feed 60 fps timestamps, target 30, assert ~half emitted); latest-wins mailbox drops intermediates (feed 3 frames while consumer blocked, assert only newest delivered).open()failure paths (nonexistent file → error signal). Real camera tests are impossible in CI — guard with aQTMESH_MOCAP_CAMERA_TESTSenv like the skin reference tests.Acceptance criteria
cmake -DENABLE_MOCAP=ON -DENABLE_ONNX=ONbuilds on macOS + Linux CI; default build is bit-identical in behavior (no new deps pulled).FileFrameSourceplays an mp4 and delivers RGB888 frames with monotonically increasing timestamps (manual verification note in the PR).ImageSequenceFrameSource+ decimation + latest-wins covered by headless unit tests.NSCameraUsageDescriptionadded toInfo.plist.in.