Isaac ROS Visual SLAM

Webinar Available

Learn how to use this package by watching our on-demand webinar: Pinpoint, 250 fps, ROS 2 Localization with vSLAM on Jetson

Overview

This repository provides a high-performance, best-in-class ROS 2 package for VSLAM (visual simultaneous localization and mapping). This package uses a stereo camera with an IMU to estimate odometry as an input to navigation. It is GPU accelerated to provide real-time, low-latency results in a robotics application. VSLAM provides an additional odometry source for mobile robots (ground based) and can be the primary odometry source for drones.

VSLAM provides a method for visually estimating the position of a robot relative to its start position, known as VO (visual odometry). This is particularly useful in environments where GPS is not available (such as indoors) or intermittent (such as urban locations with structures blocking line of sight to GPS satellites). This method is designed to use left and right stereo camera frames and an IMU (inertial measurement unit) as input. It uses input stereo image pairs to find matching key points in the left and right images; using the baseline between the left and right camera, it can estimate the distance to the key point. Using two consecutive input stereo image pairs, VSLAM can track the 3D motion of key points between the two consecutive images to estimate the 3D motion of the camera--which is then used to compute odometry as an output to navigation. Compared to the classic approach to VSLAM, this method uses GPU acceleration to find and match more key points in real-time, with fine tuning to minimize overall reprojection error.

Key points depend on distinctive features in the left and right camera image that can be repeatedly detected with changes in size, orientation, perspective, lighting, and image noise. In some instances, the number of key points may be limited or entirely absent; for example, if the camera field of view is only looking at a large solid colored wall, no key points may be detected. If there are insufficient key points, this module uses motion sensed with the IMU to provide a sensor for motion, which, when measured, can provide an estimate for odometry. This method, known as VIO (visual-inertial odometry), improves estimation performance when there is a lack of distinctive features in the scene to track motion visually.

SLAM (simultaneous localization and mapping) is built on top of VIO, creating a map of key points that can be used to determine if an area is previously seen. When VSLAM determines that an area is previously seen, it reduces uncertainty in the map estimate, which is known as loop closure. VSLAM uses a statistical approach to loop closure that is more compute efficient to provide a real time solution, improving convergence in loop closure.

There are multiple methods for estimating odometry as an input to navigation. None of these methods are perfect; each has limitations because of systematic flaws in the sensor providing measured observations, such as missing LIDAR returns absorbed by black surfaces, inaccurate wheel ticks when the wheel slips on the ground, or a lack of distinctive features in a scene limiting key points in a camera image. A practical approach to tracking odometry is to use multiple sensors with diverse methods so that systemic issues with one method can be compensated for by another method. With three separate estimates of odometry, failures in a single method can be detected, allowing for fusion of the multiple methods into a single higher quality result. VSLAM provides a vision- and IMU-based solution to estimating odometry that is different from the common practice of using LIDAR and wheel odometry. VSLAM can even be used to improve diversity, with multiple stereo cameras positioned in different directions to provide multiple, concurrent visual estimates of odometry.

To learn more about VSLAM, refer to the Elbrus SLAM documentation.

Accuracy

VSLAM is a best-in-class package with the lowest translation and rotational error as measured on KITTI Visual Odometry / SLAM Evaluation 2012 for real-time applications.

Method	Runtime	Translation	Rotation	Platform
VSLAM	0.007s	0.94%	0.0019 deg/m	Jetson AGX Xavier aarch64
ORB-SLAM2	0.06s	1.15%	0.0027 deg/m	2 cores @ >3.5 Ghz x86_64

In addition to results from standard benchmarks, we test loop closure for VSLAM on sequences of over 1000 meters, with coverage for indoor and outdoor scenes.

Performance

The following table summarizes the per-platform performance statistics of sample graphs that use this package, with links included to the full benchmark output. These benchmark configurations are taken from the Isaac ROS Benchmark collection, based on the ros2_benchmark framework.

Sample Graph	Input Size	AGX Orin	Orin NX	Orin Nano 8GB	x86_64 w/ RTX 3060 Ti
Visual SLAM Node	720p	228 fps 36 ms	124 fps 39 ms	116 fps 85 ms	238 fps 50 ms

Commercial Support & Source Access

For commercial support and access to source code, please contact NVIDIA.

Latest Update

Update 2022-03-23: Update Elbrus library and performance improvements

Supported Platforms

This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note: Versions of ROS 2 earlier than Humble are not supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.

Platform	Hardware	Software	Notes
Jetson	Jetson Orin Jetson Xavier	JetPack 5.1.1	For best performance, ensure that power settings are configured appropriately.
x86_64	NVIDIA GPU	Ubuntu 20.04+ CUDA 11.8+

Docker

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Coordinate Frames

This section describes the coordinate frames that are involved in the VisualSlamNode. The frames discussed below are oriented as follows:

input_base_frame: The name of the frame used to calculate transformation between baselink and left camera. The default value is empty (''), which means the value of base_frame_ will be used. If input_base_frame_ and base_frame_ are both empty, the left camera is assumed to be in the robot's center.
input_left_camera_frame: The frame associated with left eye of the stereo camera. Note that this is not the same as the optical frame. The default value is empty (''), which means the left camera is in the robot's center and left_pose_right will be calculated from the CameraInfo message.
input_right_camera_frame: The frame associated with the right eye of the stereo camera. Note that this is not the same as the optical frame. The default value is empty (''), which means left and right cameras have identity rotation and are horizontally aligned, so left_pose_right will be calculated from CameraInfo.
input_imu_frame: The frame associated with the IMU sensor (if available). It is used to calculate left_pose_imu.

Quickstart

Set up your development environment by following the instructions here.

Note: ${ISAAC_ROS_WS} is defined to point to either /ssd/workspaces/isaac_ros-dev/ or ~/workspaces/isaac_ros-dev/.

Clone this repository and its dependencies under ${ISAAC_ROS_WS}/src.

cd ${ISAAC_ROS_WS}/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

[Terminal 1] Launch the Docker container

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
  ./scripts/run_dev.sh ${ISAAC_ROS_WS}

[Terminal 1] Inside the container, build and source the workspace:

cd /workspaces/isaac_ros-dev && \
  colcon build --symlink-install && \
  source install/setup.bash

(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential

Run the following launch files in the current terminal (Terminal 1):

ros2 launch isaac_ros_visual_slam isaac_ros_visual_slam.launch.py

[Terminal 2] Attach another terminal to the running container for Rviz2

Attach another terminal to the running container for RViz2.
```
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
  ./scripts/run_dev.sh ${ISAAC_ROS_WS}
```
From this second terminal, run Rviz2 to display the output:
```
source /workspaces/isaac_ros-dev/install/setup.bash && \
  rviz2 -d src/isaac_ros_visual_slam/isaac_ros_visual_slam/rviz/default.cfg.rviz 
```
If you are SSH-ing in from a remote machine, the Rviz2 window should be forwarded to your remote machine.

[Terminal 3] Attach the 3rd terminal to start the rosbag

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
  ./scripts/run_dev.sh ${ISAAC_ROS_WS}

Run the rosbag file to start the demo.

source /workspaces/isaac_ros-dev/install/setup.bash && \
  ros2 bag play src/isaac_ros_visual_slam/isaac_ros_visual_slam/test/test_cases/rosbags/small_pol_test/

Result:

Rviz should start displaying the point clouds and poses like below:

Note: By default, IMU data is ignored. To use IMU data in Isaac ROS Visual Slam, please set enable_imu param in your launch file to True. See example below.

Next Steps

Try More Examples

To continue your exploration, check out the following suggested examples:

Tutorial with Isaac Sim
Tutorial for Visual SLAM using a RealSense camera with integrated IMU

Customize your Dev Environment

To customize your development environment, reference this guide.

Package Reference