Mocap Slice A — Spike: model conversion (MediaPipe face/pose, SAM 3D Body) + licensing due diligence

[fernandotonon]

Part of epic #869. Do this slice first — it de-risks everything downstream. No app code changes; deliverables are offline scripts, docs, and go/no-go decisions.

Goal

Prove the three model conversions numerically and settle the licensing questions, so Slices C/E are wiring jobs instead of research.

Task 1 — MediaPipe Face Landmarker → ONNX, with parity proof

The face_landmarker_v2_with_blendshapes.task bundle (download from Google's MediaPipe model page, Apache-2.0) is a zip containing three TFLite models:

1. Face detector (BlazeFace short-range),
2. Face landmark model (478 3D landmarks + presence/tracking scores),
3. Blendshape model (MLP-Mixer; input = a subset of ~146 2D landmarks, output = 52 blendshape scores).

Write scripts/export-facecap-onnx.py (one-time offline dev tool, NOT shipped — follow the header comment style of scripts/export-rmib-onnx.py):

• Unzip the .task, extract the three .tflite files.
• Convert each with tf2onnx (python -m tf2onnx.convert --tflite model.tflite --output model.onnx --opset 18). If tf2onnx chokes on custom ops, fall back to tflite2onnx or onnx2tf round-trip — document which worked.
• Parity test built into the script: run the Python mediapipe FaceLandmarker on 5–10 test images (include: neutral face, open mouth, wink, smile, head turned ~30°), then run the exported ONNX graphs with onnxruntime implementing the same pre/post-processing, and assert blendshape scores match within ~1e-2 and landmarks within ~1px. Print the exact pre/post-processing contract discovered (input sizes, normalization ranges, letterbox behavior, which 146 landmark indices feed the blendshape model, score/sigmoid handling, detector anchor decode) — this contract is the spec Slice C implements in C++, so write it into the doc, not just the console.
• Also extract the canonical face landmark positions MediaPipe uses for its facial transformation matrix (Procrustes reference) — needed for head pose in Slice C. If pulling MediaPipe's own geometry pipeline data is awkward, document the alternative: Umeyama rigid fit of a fixed stable-landmark subset (nose bridge, eye corners, ears region) against their positions in a reference neutral frame.

Task 2 — MediaPipe Pose Landmarker → ONNX (the body fallback)

Same recipe for pose_landmarker_full.task (detector + landmark model, 33 landmarks with world coordinates). Add to the same script (--pose flag). Parity: world-landmark positions within ~1cm on test images. Document the landmark index → body-part table (0=nose, 11/12=shoulders, 13/14=elbows, 15/16=wrists, 23/24=hips, 25/26=knees, 27/28=ankles, …) in the doc.

Task 3 — SAM 3D Body: license due-diligence + export feasibility

1. Read the SAM License (from the facebook/sam-3d-body-dinov3 HF repo and Meta's release). Answer specifically: (a) may we redistribute converted ONNX weights on our HF models repo (fernandotonon/QtMeshEditor-models)? (b) may end users use outputs commercially? (c) any acceptable-use constraints that matter for a general-purpose editor? Record the verdict + reasoning in THIRD_PARTY_AI_MODELS.md (follow the PBRify/UniRig/SkinTokens entry format). MHR (the rig definition) is Apache-2.0 — record that separately; we need MHR's joint list + rest pose regardless.
2. If the license passes: export to ONNX. Study SAM3DBody-cpp first — it already runs the model via ONNX Runtime, so its export recipe/graph layout is the shortcut. Write scripts/export-bodycap-onnx.py. Measure: model size on disk, single-frame latency on M-series CPU (target: ≤200 ms/frame is fine for offline video; live body capture can run at reduced rate). Document the I/O contract: input image size/normalization, output layout (127 MHR joint rotations — which format: 6D rotation? quaternion? Euler? — plus root translation and shape betas we'll ignore).
3. Extract the MHR skeleton definition (127 joint names, parent hierarchy, rest/bind pose world orientations) from the Apache-2.0 MHR repo into a JSON we can embed or host — Slice E's retarget needs each joint's rest world orientation for the same W · clip · W⁻¹ conjugation applyMotionClip does with cmuRestWorld.
4. If the license FAILS the bar: record the rejection in THIRD_PARTY_AI_MODELS.md (like RigNet/SF3D/Hunyuan3D) and note that Slice E ships Pose-IK only.

Task 4 — Write the spike doc

docs/MOCAP_SPIKE.md (follow docs/IMAGE_TO_3D_SPIKE_764.md): conversion recipes + versions pinned, parity numbers, pre/post-processing contracts for all models, MHR skeleton notes, license verdicts, model sizes, measured latencies, and the go/no-go recommendation for the SAM backend.

Acceptance criteria

• scripts/export-facecap-onnx.py produces detector/landmarks/blendshapes ONNX with in-script parity vs Python mediapipe (tolerances above) on the test images.
• Pose landmarker exported with the same parity treatment.
• SAM License verdict written in THIRD_PARTY_AI_MODELS.md; if PASS, scripts/export-bodycap-onnx.py + measured size/latency; if FAIL, rejection recorded and epic scope adjusted.
• MHR skeleton JSON (names, hierarchy, rest world orientations) produced.
• docs/MOCAP_SPIKE.md written with the full I/O + preprocessing contracts (the implementable spec for Slices C/E).

[fernandotonon]

Shipped in the consolidated Performance Capture epic PR #909 (merged). All slice code, pure-data tests, docs (docs/MOCAP.md, docs/MOCAP_SPIKE.md), and CLI qtmesh mocap / MCP (capture_face_from_video, capture_body_from_video, list_capture_devices, start/stop_live_capture, set_capture_channels) / GUI (Animation → Performance Capture) surfaces are on master.