Multi-object tracking (Kalman filter + Hungarian algorithm) and trajectory prediction, in pure Python.
PyRoboVision's real, tested core is a multi-object tracker: Kalman filter state estimation + Hungarian algorithm association, with occlusion handling (tracks are predicted through gaps in detection and re-associate on reacquisition) and constant velocity/acceleration trajectory prediction with uncertainty quantification.
Around that core there are supporting utilities: 3D perception (a real MiDaS monocular depth backend, 3D bounding box conversion, LiDAR point-cloud processing, occupancy grids), rule-based behavior/intent classification, and simple imitation learning / behavior cloning / safety-constraint building blocks.
This is not a full detection -> tracking -> 3D -> planning -> safety autonomous driving stack, and doesn't claim to be. See What's NOT included below for specifics — an earlier version of this README oversold this project, and we'd rather be boring and accurate than impressive and wrong.
The core package depends on NumPy and SciPy only. PyTorch is an optional extra, needed only if you want real MiDaS depth inference or the ONNX export helpers.
- Adding multi-object tracking on top of your own detector — feed
bounding boxes from YOLO/SAM/whatever into
MOTTracker; it handles identity persistence through occlusion. - Trajectory forecasting from tracked positions — constant-velocity/ acceleration prediction with uncertainty, for collision-avoidance-style logic downstream.
- IMU/GPS sensor fusion for a fused position estimate with covariance, without hand-rolling a Kalman filter.
- Not yet a good fit for: a full detection pipeline (bring your own detector), GPU-accelerated inference, metric (not just ordered) depth without your own calibration, or foundation-model integration (CLIP/SAM/ Grounding DINO — an earlier version claimed this via a module that returned hardcoded fake results; it's been removed, not shipped — see What's NOT included).
import numpy as np
from pyrobovision.tracking.mot import MOTTracker, Detection
tracker = MOTTracker(max_age=30, min_hits=3)
# Detections come from *your* detector (YOLO, SAM, a color blob detector,
# whatever) — PyRoboVision does not ship one. Each is just a bounding box
# plus a confidence score.
detections = [Detection(bbox=np.array([100, 100, 140, 180]), confidence=0.92)]
confirmed_tracks = tracker.update(detections)
for track in confirmed_tracks:
print(f"track {track.track_id}: position={track.get_position()}, velocity={track.get_velocity()}")pip install pyrobovision
# With real MiDaS depth estimation (installs PyTorch, one-time model download)
pip install "pyrobovision[depth]"
# With ONNX model export utilities
pip install "pyrobovision[onnx]"
# From source
git clone https://github.com/Mullassery/PyRoboVision.git
cd PyRoboVision
pip install -e ".[dev]"import numpy as np
from pyrobovision.tracking.mot import MOTTracker, Detection
from pyrobovision.prediction.trajectory import TrajectoryPredictor
tracker = MOTTracker(max_age=30, min_hits=3)
# Simulate a few frames of detections for one object moving right.
boxes = [
np.array([100.0, 100.0, 140.0, 180.0]),
np.array([110.0, 100.0, 150.0, 180.0]),
np.array([120.0, 100.0, 160.0, 180.0]),
np.array([130.0, 100.0, 170.0, 180.0]),
]
positions = []
for box in boxes:
tracker.update([Detection(bbox=box, confidence=0.9)])
track = tracker.get_track_by_id(1)
positions.append(track.get_position())
# Predict where the object goes next, using its tracked position history.
predictor = TrajectoryPredictor(model="cv") # "cv" = constant velocity, "ca" = constant acceleration
future_positions = predictor.predict_trajectory(np.array(positions), horizon=5)
print(future_positions)Occlusion is handled automatically: if a tracked object stops being detected for a
few frames (tracker.update([])), its Kalman filter keeps predicting its position
forward, and the track survives until max_age frames without a match — then a
detection reappearing near the predicted location re-associates to the same
track_id. This is exercised directly in
tests/test_occlusion.py.
import numpy as np
from pyrobovision.perception.depth import DepthEstimator
from pyrobovision.perception.bbox_3d import Box3DConverter
from pyrobovision.perception.occupancy import OccupancyGridBuilder
# model="heuristic" (default) needs no extra dependencies but is a
# non-neural edge-density PLACEHOLDER, not real depth — see the module
# docstring in perception/depth.py. For genuine learned depth:
# pip install "pyrobovision[depth]"
# estimator = DepthEstimator(model="midas")
estimator = DepthEstimator(model="heuristic")
estimator.set_calibration(fx=500, fy=500, cx=320, cy=240)
rgb_frame = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8) # your camera frame
depth_map = estimator.estimate_depth(rgb_frame)
converter = Box3DConverter()
bbox_3d = converter.from_2d_bbox_and_depth(
bbox_2d=np.array([100, 100, 200, 200]),
depth_map=depth_map.data,
fx=500, fy=500, cx=320, cy=240,
)
builder = OccupancyGridBuilder(grid_size=(100, 100), resolution=0.1)
occupancy_grid = builder.from_3d_bboxes([bbox_3d])
print(f"Occupied cells: {len(occupancy_grid.get_occupied_cells())}")from pyrobovision.fusion.sensor_fusion import SensorFusionEngine, IMUData, GPSData
from pyrobovision.learning.safety import SafetyValidator
fusion_engine = SensorFusionEngine(origin_lat=37.7749, origin_lon=-122.4194)
state = fusion_engine.update_imu(IMUData(timestamp=0.0, accelerometer=[0, 0, 9.8], gyroscope=[0, 0, 0]))
state = fusion_engine.update_gps(GPSData(timestamp=0.1, latitude=37.7749, longitude=-122.4194, altitude=10.0,
speed=5.0, heading=90.0, accuracy=2.0))
print(f"Fused position: {state.position}, uncertainty: {state.covariance}")
validator = SafetyValidator(max_acceleration=5.0, max_steering=45.0)
proposed_action = np.array([6.5, 50.0]) # [acceleration, steering_angle] — over both limits
safe_action = validator.correct_action(proposed_action, {"speed": 15.0})| Module | Status |
|---|---|
tracking/ (Kalman filter, Hungarian association, MOT) |
Real, tested (test_kalman_filter.py, test_association.py, test_mot_tracker.py, test_occlusion.py) |
prediction/ (trajectory forecasting, uncertainty) |
Real, tested |
perception/depth.py with model="midas" |
Real — genuine pretrained MiDaS_small inference via torch.hub (optional [depth] extra) |
perception/depth.py with model="heuristic" (default) |
Placeholder — Sobel-edge heuristic, not a depth model, documented as such in code |
perception/bbox_3d.py, lidar.py, occupancy.py |
Real geometric utilities (not neural), tested |
behavior/, intent/ |
Rule-based classification, not learned models |
learning/ (imitation, behavior cloning, safety, training) |
Real, minimal implementations; not benchmarked against production RL frameworks |
fusion/sensor_fusion.py |
Real Kalman-based IMU/GPS fusion |
fusion/optimization.py (ONNX export) |
Real, thin wrapper around torch.onnx.export (optional [onnx]/[depth] extra) |
- No object detector. You bring your own detections (bounding boxes + confidence)
into
MOTTracker. There is no YOLO/SAM/etc. bundled or wrapped here. - No GPU-accelerated inference pipeline. The tracking/prediction core is plain NumPy/SciPy; it runs on CPU.
- No foundation-model integration (CLIP, SAM, Grounding DINO). An earlier version
of this package shipped an "MCP 2.0" module and CLI/server "workflow" layer that
claimed to do this — it only ever returned hardcoded fake results (e.g. a fixed
"frames_processed": 1200regardless of input) and has been removed. - Not a certified or safety-verified autonomous driving stack.
SafetyValidatoris a basic constraint-checking utility, not a certified safety system. - MiDaS depth output is relative, not metric.
model="midas"gives correctly ordered near/far depth, not depth in meters, unless you calibrate it against a known reference distance yourself.
pip install -e ".[dev]"
pytest tests/ -v276 passing / 1 skipped, 87% coverage (measured for real - --cov=src/pyrobovision,
1895 statements / 244 missed; the previous --cov=pyrobovision config used the package
name instead of a path, which silently reported "no data collected" rather than a real
number - the 89% this README used to claim was never actually measured), on Python
3.11 (no torch installed — the one test that requires real MiDaS inference skips
cleanly without it and runs when the [depth] extra is installed). CI runs the same
command on every push.
src/pyrobovision/
├── tracking/ Kalman filter + Hungarian-algorithm multi-object tracking
├── prediction/ Trajectory forecasting (CV/CA models) + uncertainty
├── perception/ Depth estimation, 3D bbox conversion, LiDAR, occupancy grids
├── behavior/ Rule-based motion/behavior classification
├── intent/ Rule-based intent prediction
├── learning/ Imitation learning, behavior cloning, safety constraints, training loop
└── fusion/ IMU/GPS sensor fusion, ONNX export utilities
- PyRoboFrames — a separate, independently published dataloader project by the same author (LeRobot, RLDS, HDF5, NetCDF, S3-compatible object storage). PyRoboVision does not depend on it — if you want to feed PyRoboFrames-loaded data into this tracker, install it separately.
Cross-repo compatibility, whole family: this author also publishes
PyRoboSimulator, pyroboreplay, and PyTerrainMap. Verified by reading
every Cargo.toml/pyproject.toml across all of them: none has a Cargo or
pip dependency on PyRoboVision, and PyRoboVision has none on them — each is
independently installable and versioned.
- CONTRIBUTING.md — development setup and guidelines
- ROADMAP.md — what's done, what's not, what's next
- SECURITY.md — vulnerability reporting
- docs/CODE_OF_CONDUCT.md — community guidelines
- docs/BENCHMARKS.md — current benchmark status (none published yet — how to run your own)
Contributions welcome! See CONTRIBUTING.md.
For security issues, see SECURITY.md.
This project is licensed under the Apache License 2.0.
@software{mullassery2026pyrobovision,
title={PyRoboVision: Multi-object tracking and trajectory prediction for robotics},
author={Mullassery, Georgi},
url={https://github.com/Mullassery/PyRoboVision},
year={2026}
}