Epic: AI — Performance capture: video/webcam → facial morph + skeletal body animation (live preview + record)

[fernandotonon]

Overview

Record a video of yourself — or point a webcam at yourself live — and reproduce the performance on a mesh in the editor:

• Face: per-frame facial expression → 52 ARKit-style blendshape weights → morph-target weight keyframes on the entity (the exact (time, target, weight) stream that MorphAnimationManager::setMorphWeightKeyframe already consumes — Anim: Slice B — Mesh / vertex animation (Alembic + Ogre VAT_POSE clips) #519 shipped the whole downstream pipeline: authoring, dope-sheet diamonds, timeline playback, glTF morph-weight-animation export).
• Head: per-frame head pose (rotation + translation) → keyframes on the Head/Neck bone (skinned mesh) or node TRS clip (static mesh, via NodeAnimationManager).
• Body: per-frame full-body pose → per-joint rotations → retargeted onto the user's rig through the existing 22-joint canonical mapping + world-frame retarget (MotionInbetween::canonicalIndexForBone + the AnimationMerger::applyMotionClip conjugation math — AI: Animation in-betweening (ONNX) #409/AI: Text-to-motion via MDM (ONNX) — research/ambitious #411 built this for exactly this shape of problem).
• Live mode: webcam preview driving the mesh in real time (morph weights + manually-controlled bones), with Record writing the take into an ordinary animation clip that plays on the existing timeline and exports through the existing exporters.

This is "markerless performance capture" — the VTuber / Rokoko-Video / Move.ai feature class — built on infrastructure this codebase already has: ONNX Runtime (8 consumers today), ModelDownloader + HF hosting, canonical-skeleton retargeting, morph weight clips, worker-thread AI controllers.

Model selection & licensing (the deciding factor, as always)

Face — MediaPipe Face Landmarker (Google) — Apache-2.0 code AND models ✅

• 3-stage pipeline: BlazeFace face detector → 3D face landmark model (478 landmarks) → blendshape head (small MLP-Mixer) emitting 52 ARKit-compatible blendshape coefficients (jawOpen, mouthSmileLeft, eyeBlinkRight, browInnerUp, …) + a facial transformation matrix (head pose for free).
• Ships as TFLite inside a .task bundle — not ONNX. The offline conversion (tf2onnx / tflite2onnx; PINTO model zoo has proven these models convert) is a first-class spike deliverable, exactly like scripts/export-rmib-onnx.py / export-triposr-onnx.py. Models are small (detector + landmarks + blendshapes ≈ a few MB total) — trivial hosting/download.
• This is what the entire VTuber ecosystem runs on; quality is proven for this use case.
• Rejected: DECA / EMOCA / SPECTRE (all regress the FLAME 3DMM — FLAME is research-only; the LAFAN1/AMASS/ShapeNet wall again), ARKit (iOS-only), OpenSeeFace (MIT code but weaker blendshape story).

Body — two backends, quality path + always-available permissive fallback (the SkinTokens/GeodesicVoxel house pattern)

1. SAM 3D Body + MHR (Meta, Nov 2025) — the quality path. Regresses full-body pose as per-joint rotations on the 127-joint Momentum Human Rig from a single image. MHR itself is Apache-2.0 (explicitly commercially licensed — Meta's SMPL replacement); the SAM 3D Body model checkpoints are under the "SAM License" (permissive with conditions + acceptable-use policy). Slice A must do the license due-diligence read (redistribution/rehosting terms for our HF models repo) and record the decision in THIRD_PARTY_AI_MODELS.md — the PBRify/UniRig precedent. A community pure-C++ ONNX port exists (SAM3DBody-cpp) proving the export + runtime path is viable. Per-image model → video = per-frame inference + temporal smoothing.
2. MediaPipe Pose Landmarker (Apache-2.0) + analytic IK — the fallback, always compiled. 33 3D world landmarks per frame; we solve bone rotations from landmark segment directions vs the rig's bind directions (the same change-of-basis conjugation applyMotionClip already does). Softer quality, tiny models, zero license risk. Used automatically when the SAM model is unavailable / un-downloadable / fails — algorithmUsed + fallbackReason in the report, house pattern.

• Rejected: WHAM / GVHMR / TRAM / 4D-Humans (all regress SMPL, whose weights are non-commercial — Meshcapade sells the commercial license; the recurring wall), OpenPose (CMU non-commercial), FreeMoCap (AGPL).

Related but out of scope here

NVIDIA Audio2Face (open-sourced 2025) — audio → blendshapes, no camera. Would ride the same morph-weight-keyframe plumbing; belongs in its own follow-up epic.

The real risks (read before scheduling)

1. Qt Multimedia is a new dependency (camera + video decode). Qt 6.5+ ships an FFmpeg backend in the official binaries, but CI (aqt module list), the Debian package depends, the macOS bundle, and MinGW all need verification. Mitigate: everything lives behind a new ENABLE_MOCAP CMake flag (default OFF; ON for release builds once CI is green on all three platforms) — the ENABLE_ALEMBIC / ENABLE_ONNX precedent. A non-mocap build must print a clear "rebuild with -DENABLE_MOCAP" on every surface.
2. TFLite → ONNX conversion parity for the MediaPipe models is assumed, not proven — Slice A validates numerically against the Python mediapipe reference before anything else is built.
3. SAM License may or may not permit rehosting the converted ONNX on our HF repo — Slice A decides; if it fails the bar, the epic still ships with MediaPipe-Pose-IK as the only body backend (feature degrades, doesn't die).
4. Live-drive of a skinned mesh requires Bone::setManuallyControlled(true) during preview and a clean restore after — get this wrong and the entity's existing animations break. The preview path must snapshot + restore like UvUnwrap's GUI-safe entry point does.
5. macOS camera permission: NSCameraUsageDescription in Info.plist.in + graceful denied-permission UX.

Existing patterns to clone (do NOT rewrite)

• Predictor template: src/ImageTo3D/MeshGenPredictor / src/SkinTokensPredictor — ENABLE_ONNX guards, Ort::Session setup (CoreML EP on macOS), runtime I/O-name/shape discovery (never hardcode tensor names), ensureModelBlocking() + QTMESH_*_MODEL_BASE_URL / QTMESH_*_NO_DOWNLOAD env overrides, models under AppData/ai_models/<feature>/.
• Worker-thread AI controller: src/ImageTo3D/MeshGenController — QML_SINGLETON, inference on a worker thread, results marshalled to the main thread, progress + cancel.
• Canonical skeleton mapping + retarget: MotionInbetween::canonicalIndexForBone() (22 CMU roles, Mixamo/generic/CMU name handling) and AnimationMerger::applyMotionClip (world-frame conjugation local(f) = parentWorld⁻¹ · (W · clip(f) · W⁻¹) · parentWorld · bind, standing-pose bind harvest, locked root, rotation-only keys). Body capture output must flow through this, not a new retargeter.
• Morph weight keyframes: MorphAnimationManager::setMorphWeightKeyframe(name, time, weight) + the multi-clip morph weight support (Anim: Slice B — Mesh / vertex animation (Alembic + Ogre VAT_POSE clips) #519). The dope sheet, timeline, and glTF morph-weight-anim export all already work downstream of this call.
• Undoable bulk edit: ComputeSkinWeightsCommand snapshot pattern — a recorded take pushes ONE undo command.
• CLI/MCP surface: CLIPipeline::run dispatcher + cmdAnim/cmdSegment as reference subcommand implementations; MCPServer::callTool map + heavy-tool registration.
• Offline export scripts: scripts/export-rmib-onnx.py, scripts/export-triposr-onnx.py, scripts/upload-triposr-models.sh — one-time dev tools, NOT shipped.

Proposed architecture (new feature folder src/Mocap/)

src/Mocap/
  VideoFrameSource.{h,cpp}      # Qt Multimedia abstraction: file / camera / image-sequence (tests)
  FaceCapPredictor.{h,cpp}      # ONNX consumer #9: frame → 52 weights + head pose + confidence
  BodyCapPredictor.{h,cpp}      # ONNX consumer #10: frame → canonical joint rotations (SAM3DBody)
  PoseIKSolver.{h,cpp}          # pure-data fallback: 33 landmarks → bone rotations
  FaceCapMapper.{h,cpp}         # pure-data: ARKit-52 names → mesh morph target names
  OneEuroFilter.{h,cpp}         # pure-data smoothing (per-channel; quat variant)
  MocapRecorder.{h,cpp}         # sample stream → keyframes (morph clip + bone/node tracks)
  MocapController.{h,cpp}       # QML_SINGLETON, worker thread, live preview + record state machine
qml/PerformanceCapturePanel.qml # Animation Mode → Mode Tools section

Everything except MocapController/MocapRecorder is Ogre-free and unit-testable headless (the PbrMapSynth/MotionLibrary bar).

Surfaces

• CLI: qtmesh mocap <video> --face [--body] [--mesh <file>] [--clip-name NAME] [--fps N] [--smooth-cutoff HZ] [--algo sam3dbody|pose-ik] [--no-model] [--json] -o out.glb
• MCP: capture_face_from_video, capture_body_from_video (both heavy, file-in/scene-or-file-out), list_capture_devices, start_live_capture / stop_live_capture (GUI-attached sessions only).
• GUI: Animation Mode → Mode Tools → "Performance Capture" section — camera/device picker, live preview, Face/Body/Head toggles, channel-mapping review, Preview (drive live) and Record (write keyframes) buttons.

Child issues (slices)

• Mocap Slice A — Spike: model conversion (MediaPipe face/pose, SAM 3D Body) + licensing due diligence #870 — Slice A — Spike: model conversion + licensing due diligence (de-risk first)
• Mocap Slice B — ENABLE_MOCAP build infra + VideoFrameSource (Qt Multimedia: file / camera / image-sequence) #871 — Slice B — ENABLE_MOCAP build infra + VideoFrameSource (Qt Multimedia)
• Mocap Slice C — FaceCapPredictor (ONNX) + pure-data core (blendshape mapper, head-pose solve, One-Euro) #872 — Slice C — FaceCapPredictor + pure-data core (mapper, head pose, smoothing)
• Mocap Slice D — Face video → morph weight clip: MocapRecorder + CLI 'qtmesh mocap --face' + MCP #873 — Slice D — Face → morph clip integration + CLI/MCP
• Mocap Slice E — Body capture: SAM3DBody backend + Pose-IK fallback + canonical retarget (--body) #874 — Slice E — Body capture: SAM3DBody backend + Pose-IK fallback + retarget
• Mocap Slice F — Live camera Performance Capture panel: preview drive + record-to-clip (GUI) #875 — Slice F — Live camera GUI: Performance Capture panel (preview + record)
• Mocap Slice G — Model hosting, docs, packaging, CI, telemetry #876 — Slice G — Model hosting, docs, packaging, CI, telemetry

Recommended order: A → B → C → D (face ships end-to-end) → E (body) → F (live) → G (hardening). D is the first user-visible milestone; F is the headline feature.

Acceptance criteria

• qtmesh mocap talk.mp4 --face --mesh face_with_arkit_targets.glb -o out.glb produces a glTF whose morph-weight animation visibly reproduces the recorded speech/expressions (verify via the existing timeline playback + glTF reimport).
• qtmesh mocap dance.mp4 --body --mesh rigged.fbx -o out.glb produces a skeletal clip on the user's rig; --algo pose-ik and --no-model force the fallback; report carries algorithmUsed/fallbackReason.
• GUI live mode: webcam preview drives the selected entity's morphs + head in real time at ≥15 fps on an M-series laptop; Record produces a clip that plays on the existing timeline, is undoable (single Ctrl+Z), and exports.
• Head pose lands on the Head bone for skinned meshes and as node TRS for static meshes.
• Non-ENABLE_MOCAP builds show a clear "rebuild with -DENABLE_MOCAP" on every surface (no crash); ENABLE_MOCAP without ONNX models degrades per-surface with clear messages; camera-permission denial is handled gracefully.
• Unmatched blendshape channels are REPORTED (count + names), never silently dropped; a JSON mapping override file is honoured.
• All pure-data pieces (mapper, One-Euro, head-pose solve, Pose-IK, letterbox math) have headless unit tests; ONNX-dependent tests skip gracefully without models (UniRig/PBR test pattern).
• Licensing decisions recorded in THIRD_PARTY_AI_MODELS.md; models hosted on the HF repo (or documented "not yet hosted" state with graceful degradation).
• Sentry breadcrumbs (ai.assist.mocap_face, ai.assist.mocap_body, ui.action on panel controls) + gamification noteFeature/noteOperation instrumentation.

Out of scope

• Audio-driven facial animation (NVIDIA Audio2Face) — follow-up epic, same morph plumbing.
• Multi-person capture, finger/hand articulation fidelity, gaze/tongue tracking, facial wrinkle/texture maps.
• Foot-contact locking / ground-plane solve for body capture (accept some foot slide in v1; root is locked like AI: Text-to-motion via MDM (ONNX) — research/ambitious #411).
• Authoring the ARKit-52 morph targets themselves (the Anim: Slice B — Mesh / vertex animation (Alembic + Ogre VAT_POSE clips) #519 morph authoring UX covers manual creation; auto-generating them from a neutral mesh is its own ML problem).
• Real-time streaming out of the app (VMC/OSC protocol) — possible follow-up.