Real-time marker-based Augmented Reality built with Python, OpenCV, NumPy, ORB feature detection, RANSAC homography estimation, 3D projection, and Kalman filtering.
VisionAR detects a reference image through a webcam, estimates its perspective and orientation, and renders a 3D OBJ model on top of the tracked surface in real time.
The system includes Kalman-filter-based tracking stabilization, RANSAC confidence estimation, real-time performance monitoring, configurable CLI controls, automated testing, and GitHub Actions continuous integration.
- Real-time webcam-based augmented reality
- ORB feature detection and description
- Descriptor-based feature matching
- RANSAC-based outlier rejection
- Homography estimation
- Perspective-aware marker tracking
- Camera projection estimation
- Custom OBJ model loading
- Real-time 3D model rendering
- Kalman-filter-based corner stabilization
- Temporary tracking prediction when detection is lost
- RANSAC inlier-based confidence estimation
- Real-time FPS monitoring
- Marker boundary visualization
- ORB feature-match visualization
- Configurable command-line interface
- Automated unit tests with pytest
- GitHub Actions continuous integration
VisionAR detects the reference marker from the webcam feed, estimates its perspective, stabilizes the detected marker corners using a Kalman filter, and projects a 3D OBJ model onto the tracked surface.
The real-time interface displays:
- Current tracking status
- ORB feature-match count
- RANSAC inlier count
- Tracking confidence
- Processing FPS
- Optional marker boundary
Run the AR tracker with the marker boundary enabled:
python src/main.py --rectangle --scale 4VisionAR includes a debugging mode that visualizes feature correspondences between the reference marker and the live webcam frame.
Run feature-match visualization:
python src/main.py --matchesRun the complete debugging view:
python src/main.py --matches --rectangle --scale 4This visualization helps inspect whether ORB is identifying reliable correspondences before RANSAC estimates the marker homography.
Reference Marker
|
v
ORB Feature Detection
|
v
Feature Descriptors
|
+----------------------+
|
Webcam Frame |
| |
v |
ORB Feature Detection |
| |
v |
Feature Matching <-----------+
|
v
RANSAC
|
v
Homography Estimation
|
v
Marker Corner Detection
|
v
Kalman Filter
(Predict + Correct)
|
v
Smoothed Marker Corners
|
v
Filtered Homography
|
v
Camera Projection Matrix
|
v
3D OBJ Projection
|
v
Augmented Reality Frame
| Technology | Purpose |
|---|---|
| Python | Core application |
| OpenCV | Computer vision, feature detection, and rendering |
| NumPy | Matrix and numerical operations |
| ORB | Feature detection and binary descriptors |
| RANSAC | Robust homography estimation and outlier rejection |
| Kalman Filter | Marker tracking stabilization and prediction |
| OBJ | 3D model representation |
| pytest | Automated testing |
| GitHub Actions | Continuous integration |
VisionAR/
|
├── .github/
│ └── workflows/
│ └── tests.yml
|
├── assets/
│ ├── markers/
│ │ └── marker.png
│ │
│ └── models/
│ └── model.obj
|
├── docs/
│ └── demo/
│ ├── tracking.png
│ └── matches.png
|
├── src/
│ ├── detector.py
│ ├── homography.py
│ ├── kalman.py
│ ├── main.py
│ ├── matcher.py
│ ├── obj_loader.py
│ ├── pose_estimator.py
│ ├── projection.py
│ └── renderer.py
|
├── tests/
│ ├── test_homography.py
│ ├── test_kalman.py
│ └── test_obj_loader.py
|
├── requirements.txt
└── README.md
git clone https://github.com/saxenap2804/VisionAR.git
cd VisionARpython -m venv .venv
.venv\Scripts\activatepython3 -m venv .venv
source .venv/bin/activatepip install -r requirements.txtRun the application using the default marker, model, scale, and webcam:
python src/main.pyPress Q while the AR window is active to exit.
python src/main.py --rectanglepython src/main.py --matchespython src/main.py --scale 5python src/main.py --matches --rectangle --scale 4python src/main.py --marker path/to/marker.pngpython src/main.py --model path/to/model.objpython src/main.py --camera 1python src/main.py --help| Option | Description | Default |
|---|---|---|
--marker |
Path to the reference marker image | assets/markers/marker.png |
--model |
Path to the OBJ model | assets/models/model.obj |
--scale |
Scale factor for the rendered model | 3.0 |
--camera |
Camera device index | 0 |
--rectangle |
Display tracked marker boundary | Disabled |
--matches |
Display ORB feature correspondences | Disabled |
-h, --help |
Display command-line help | — |
VisionAR first detects distinctive keypoints in the reference marker using ORB (Oriented FAST and Rotated BRIEF).
ORB generates binary descriptors around those keypoints so they can be efficiently compared with features detected in incoming webcam frames.
The reference marker only needs to be processed once when the application starts.
For every webcam frame, VisionAR detects another set of ORB features.
Descriptors from the reference marker are compared with descriptors from the current frame to identify candidate feature correspondences.
The strongest matches are passed to the geometric estimation stage.
Feature matching can produce incorrect correspondences.
VisionAR therefore uses RANSAC during homography estimation to reject geometrically inconsistent matches.
The resulting homography describes the perspective transformation between the original planar marker and its detected position in the webcam frame.
This allows VisionAR to determine where the marker's four corners appear in the live image.
Raw frame-by-frame marker detections can introduce visible jitter.
VisionAR uses a constant-velocity Kalman filter to track the eight coordinates representing the four marker corners:
x1, y1
x2, y2
x3, y3
x4, y4
The complete Kalman state contains both position and velocity:
[x1, y1, x2, y2, x3, y3, x4, y4,
vx1, vy1, vx2, vy2, vx3, vy3, vx4, vy4]
Each frame performs:
Previous State
|
v
Prediction
|
v
Current Marker Measurement
|
v
Correction
|
v
Smoothed Corner Positions
The corrected corner coordinates are then used to calculate a new filtered homography.
If marker detection is temporarily lost, the Kalman motion model can continue predicting the marker's location for short periods.
VisionAR constructs an approximate camera intrinsic matrix using the dimensions of the webcam frame.
The filtered homography and camera matrix are then used to calculate the projection matrix required for the AR rendering stage.
VisionAR includes a custom OBJ loader.
The loader reads:
- 3D vertex coordinates
- Polygon face definitions
- Face-to-vertex relationships
This allows standard .obj geometry to be loaded without requiring a complete external 3D engine.
The OBJ vertices are transformed using the calculated projection matrix.
The resulting 2D coordinates are projected onto the webcam image, and the model faces are rendered directly using OpenCV.
The final result is a 3D object visually anchored to the tracked reference marker.
VisionAR displays tracking and performance information while the application is running.
Example:
Kalman tracking active
Matches: 47 | Inliers: 36
FPS: 29.8
Confidence: 76%
The number of feature correspondences found between the reference marker and the current webcam frame.
The number of candidate matches accepted by RANSAC as geometrically consistent with the estimated homography.
The confidence indicator is calculated using the RANSAC inlier ratio:
confidence = number of inliers / number of evaluated matches
For example:
36 inliers / 47 evaluated matches ≈ 76%
The displayed value is intended as a practical tracking-quality indicator rather than a statistically calibrated probability.
FPS reports the approximate real-time processing throughput of the computer-vision pipeline.
The displayed value is smoothed between frames to reduce visual fluctuation.
VisionAR can display different tracking states.
The marker was successfully detected in the current frame.
The Kalman filter performs both prediction and correction using the new measurement.
The marker was not successfully detected in the current frame, but the Kalman filter has previously been initialized.
The system temporarily predicts the marker location using the estimated motion state.
The marker has not yet been successfully detected and the Kalman filter cannot provide a prediction.
VisionAR includes automated tests for core mathematical, tracking, and model-loading components.
Run all tests:
pytest tests -vThe current test suite contains 12 automated tests.
Tests verify:
- Filter initialization
- Prediction output
- Correction toward new measurements
- Reset behavior
Tests verify:
- Identity homography
- Translation transformations
- Insufficient-match handling
Tests verify:
- OBJ model loading
- Vertex count
- Face count
- 3D vertex structure
- Valid face indices
Expected result:
12 passed
VisionAR uses GitHub Actions for continuous integration.
The workflow automatically installs the project dependencies and executes the pytest suite whenever code is:
- pushed to
main - submitted through a pull request targeting
main
Workflow:
Git Push / Pull Request
|
v
GitHub Actions
|
v
Python Environment
|
v
Install Dependencies
|
v
Run pytest
|
v
12 Automated Tests
The status badge at the top of this README reflects the current CI result.
VisionAR was developed around several engineering goals:
Real-time performance
The tracking pipeline needs to operate continuously on webcam frames.
Robust feature tracking
RANSAC reduces the effect of incorrect feature correspondences.
Stable augmentation
Kalman filtering reduces visible frame-to-frame jitter.
Modularity
Feature detection, matching, homography estimation, tracking, projection, OBJ loading, and rendering are separated into independent modules.
Configurability
Markers, models, camera devices, rendering scale, and debugging visualizations can be controlled through CLI arguments.
Testability
Core mathematical and parsing components can be tested independently of the webcam.
The current implementation is a lightweight AR proof of concept and has several known limitations:
- Camera intrinsics are approximated rather than obtained through full camera calibration.
- Tracking depends on a sufficiently feature-rich planar reference marker.
- Extreme motion blur can reduce ORB matching quality.
- Significant occlusion can interrupt marker detection.
- The renderer provides basic OBJ geometry rendering rather than a complete 3D graphics pipeline.
- Kalman prediction is designed primarily for short detection interruptions.
- Lighting and texture rendering are intentionally minimal.
Potential extensions include:
- Camera calibration using measured intrinsic parameters
- Multi-marker tracking
- Markerless feature tracking
- Texture-mapped OBJ models
- Improved lighting and shading
- GPU-accelerated feature processing
- Automatic 3D model selection
- Improved occlusion handling
- Full 6-DoF pose smoothing
- Better tracking-loss recovery
- AR scene recording
- Additional detector and matcher benchmarks
- Expanded automated test coverage
Core VisionAR pipeline:
- Webcam capture
- ORB feature detection
- Feature matching
- RANSAC outlier rejection
- Homography estimation
- Marker tracking
- Camera projection
- OBJ loading
- Real-time 3D rendering
- Kalman stabilization
- Tracking-loss prediction
- FPS monitoring
- RANSAC confidence indicator
- CLI configuration
- Feature-match debugging
- Automated tests
- GitHub Actions CI
- Demo screenshots
No software license has been specified yet.
Priyanka Saxena
M.S. Computer Science
Texas Tech University
GitHub: saxenap2804