Caliscope is a GUI-based multicamera calibration package. It simplifies the process of determining camera properties to enable 3D motion capture.
ballet_demo.mp4
Caliscope was used to calibrate the cameras (both intrinsic and extrinsic parameters) and triangulate the 3D landmark positions shown above.
The 2D landmark estimation was run across all videos using Google's Holistic Mediapipe.
Basic installation instructions can be found in our docs. Please note that installation can take a while due to large dependencies like OpenCV and PySide6.
For a complete overview of the entire workflow, please see the sample project. A video walk through demonstrates the process with an example dataset.
To triangulate 3D landmarks from synchronized video, you must know the intrinsic and extrinsic properties of your camera system. Intrinsic properties include each camera's focal length, optical center, and lens distortion. Extrinsic properties describe the relative rotation and translation of all cameras in the system. Using more cameras makes 3D tracking more robust to occlusion and other inevitable errors.
While OpenCV provides functions for single-camera intrinsics, estimating extrinsics for more than two cameras is not straightforward. This multi-camera process requires a technique called bundle adjustment, which demands extensive tracking of camera parameters and 2D point estimates.
Caliscope automates this calibration process from only raw video and a definition of your calibration board. It provides visual feedback at each stage, helping you verify the parameter estimates.
- Easy creation of
pngfiles for ChArUco calibration boards. - Automated calculation of camera intrinsic properties from input video.
- Visualization of the distortion model to ensure reasonableness.
- Automated bundle adjustment to estimate the 6-DoF relative position of all cameras.
- A 3D visualizer to inspect camera position estimates.
- Tools to set the world origin within the visualizer to simplify data processing.
- A general Tracker interface for integrating alternate 2D tracking methods.
- Three sample implementations using Google Mediapipe (Hands/Pose/Holistic).
- Automated application of 2D landmark tracking to synchronized videos.
- Triangulation of 3D landmark positions based on the full camera system calibration.
- Trajectory smoothing through gap-filling and Butterworth filtering.
- Output to the
.trcfile format for use in biomechanical modeling. - Output to a tidy
.csvformat for integration with other analysis workflows.
The current tracker implementations provide a proof-of-concept pipeline using Google's Mediapipe. While Mediapipe is an easy and efficient method for human tracking, it has limitations in accuracy and precision. The planned roadmap includes integration with more powerful tools like MMPose, DeepLabCut, and SLEAP.
To report a bug or request a feature, please open an issue. Please keep in mind this is an open-source project supported by volunteer effort, so your patience is appreciated.
For general questions and conversation, please post in the Discussions section of the repo.
This project was inspired by FreeMoCap (FMC), which is spearheaded by Jon Matthis, PhD of the HuMoN Research Lab. The FMC calibration and triangulation system is built upon Anipose, created by Lili Karushchek, PhD. Caliscope was originally envisioned as an alternative calibration tool to Anipose that would allow more granular estimation and visual feedback.
Several lines of the original Anipose triangulation code are used in this code base, though it was otherwise written from the ground up. I'm grateful to Dr. Matthis for his time developing FreeMoCap, discussing it with me, and providing a great deal of information regarding open-source project management.
Caliscope is licensed under the permissive BSD 2-Clause license. The triangulation function was adapted from the Anipose code base which is also licensed under the BSD-2 Clause. A primary dependency of this project is PySide6 which provides the GUI front end. PySide6 is licensed under the LGPLv3. Caliscope does not modify the underlying source code of PySide6 which is available via PyPI.