Autonomous Ackermann Steering ROS2 Robot (Simulation + Real Hardware)

An autonomous Ackermann-steered robot, from low-level hardware control to high-level autonomy powered by an LLM agent with voice input. The full pipeline runs identically in Gazebo Harmonic simulation and on a Raspberry Pi–based physical robot.

Project_Demo.mp4

---

Features

• Custom ros2_control Hardware Interface — Serial (UART) for drive motors, encoders & IMU; I2C for steering servo.
• Modular URDF / Xacro — Full Ackermann kinematic model with STL meshes and conditional sim/hardware plugin loading.
• Sensor Fusion (EKF) — robot_localization fusing wheel odometry and IMU for drift-corrected odometry.
• SLAM & Localization — SLAM Toolbox for mapping unknown environments; AMCL for particle-filter localization on saved maps.
• Nav2 Navigation — SmacPlannerHybrid (Reeds-Shepp) + MPPI Controller with Ackermann motion model and custom Behavior Tree.
• Gazebo Harmonic Simulation — Full sim environment with gz_ros2_control, simulated LiDAR and camera.
• Vision-Language Detection (Hardware Only) — Object detection with Grounding DINO for natural-language object search.
• Agentic LLM Control — Conversational AI agent (supports cloud via Gemini API or local models via Ollama) with tool-calling for navigation, camera queries, spatial memory, and location persistence.
• Web Interface with Voice Input — Browser-based chat UI served over HTTPS with OpenAI Whisper speech-to-text for hands-free voice commands.

---

System Overview

---

System Requirements

This project has been developed and tested on the following stack:

Component	Version
OS	Ubuntu 22.04
ROS 2	Humble
DDS Middleware	Cyclone DDS
Simulator	Gazebo Harmonic

Cyclone DDS

For best performance on wireless networks with large data packets (LiDAR scans, camera images), it is recommended to use Cyclone DDS instead of the default RMW.

sudo apt install -y ros-humble-rmw-cyclonedds-cpp
echo "export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp" >> ~/.bashrc
source ~/.bashrc

Gazebo Harmonic

ROS 2 Humble defaults to Gazebo Fortress. To use Gazebo Harmonic, follow the official guide for installing Gazebo Harmonic with ROS 2 Humble:

🔗 Gazebo Harmonic Installation with ROS 2 Humble

---

Installation

Dev Machine / Simulation

Note: Make sure Gazebo Harmonic is installed and sourced before building the workspace. Otherwise, rosdep might install the wrong gazebo version.

mkdir -p ~/ackermann_ws/src
cd ~/ackermann_ws/src
git clone https://github.com/sseifsalama/ackermann_robot.git

# Clone gz_ros2_control from source
git clone https://github.com/ros-controls/gz_ros2_control.git -b humble

# Install all ROS dependencies
cd ~/ackermann_ws
rosdep update
rosdep install -y -r -q --from-paths src --ignore-src --rosdistro humble

# Build
colcon build --symlink-install
source install/setup.bash

Generate SSL certificates (required for the web agent UI — browsers need HTTPS to allow microphone access over LAN):

mkdir -p ~/ackermann_ws/src/ackermann_robot/ackermann_agent/ackermann_agent/certs
cd ~/ackermann_ws/src/ackermann_robot/ackermann_agent/ackermann_agent/certs

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes \
  -subj "/CN=$(hostname).local"

Robot (Raspberry Pi / ARM)

On ARM-based boards, some packages need to be built from source and heavy packages should be excluded.

1. Exclude simulation & AI packages (not needed on-board):

cd ~/ackermann_ws/src
touch ackermann_robot/ackermann_agent/COLCON_IGNORE
touch ackermann_robot/ackermann_simulation/COLCON_IGNORE

2. Build Nav2 from source (For ARM-based boards):

cd ~/ackermann_ws/src
git clone https://github.com/ros-planning/navigation2.git -b humble

cd ~/ackermann_ws
rosdep install -y -r -q --from-paths src --ignore-src --rosdistro humble
colcon build --symlink-install --packages-up-to navigation2

3. Build RPLidar driver from source:

cd ~/ackermann_ws/src
git clone https://github.com/Slamtec/sllidar_ros2.git

4. Install remaining dependencies and build:

cd ~/ackermann_ws
rosdep update
rosdep install -y -r -q --from-paths src --ignore-src --rosdistro humble
colcon build --symlink-install
source install/setup.bash

---

Running — Simulation

1. Launch the simulated robot (Gazebo + ros2_control + RViz):

ros2 launch ackermann_simulation sim.launch.py

2. SLAM or AMCL:

# New environment
ros2 launch ackermann_bringup slam.launch.py use_sim_time:=true

# Known map
ros2 launch ackermann_bringup amcl.launch.py map:=/path/to/my_map.yaml use_sim_time:=true

3. Navigation (Nav2):

ros2 launch ackermann_bringup nav2.launch.py use_sim_time:=true

4. Autonomy Stack:

# LLM Agent
export GEMINI_API_KEY=your_api_key

# Opens at https://localhost:5000 — web UI with voice input (Whisper)
ros2 run ackermann_agent web_agent

# Alternative: LLM Agent — CLI mode
ros2 run ackermann_agent terminal_agent

---

Running — Real Robot

1. Launch the hardware interface (ros2_control + robot_state_publisher):

ros2 launch ackermann_bringup robot.launch.py

2. Launch the LiDAR and Camera:

ros2 launch ackermann_bringup lidar.launch.py

# OPTIONAL: Camera (for Grounding DINO)
ros2 launch ackermann_camera camera.launch.py

3. SLAM or AMCL:

# New environment
ros2 launch ackermann_bringup slam.launch.py

# Known map (Open rviz on dev machine and set pose)
ros2 launch ackermann_bringup amcl.launch.py map:=/path/to/my_map.yaml

4. Navigation:

ros2 launch ackermann_bringup nav2.launch.py

5. Autonomy Stack (On Dev Machine):

# OPTIONAL: Grounding DINO vision service (needs GPU)
ros2 run ackermann_agent grounding_dino

# LLM Agent
export GEMINI_API_KEY=your_api_key

# Opens at https://localhost:5000 — web UI with voice input (Whisper)
ros2 run ackermann_agent web_agent

# Alternative:LLM Agent — CLI mode
ros2 run ackermann_agent terminal_agent

---

Configuring the LLM Agent

The AI agent uses the google-genai Python library to interact with cloud models (like Google Gemini or Gemma) and also supports fully local inference via ollama.

If you want to edit the default model types or toggle between local/cloud processing, you can edit the initialization code in both ackermann_agent/agent_web.py and ackermann_agent/agent_terminal.py:

# Enable local Ollama models by changing False to True
self.declare_parameter('use_local_model', False)

# Change which Ollama model is called
self.declare_parameter('local_model_name', 'gemma4:e2b')

# Change which Cloud GenAI model is used
self.chat = self.client.chats.create(model='gemma-3-27b-it')

---

Docker

Pre-built Dockerfiles are provided in the docker/ directory as an alternative to manual installation.

Simulation / Dev Machine

To be implemented

Robot Hardware

To be implemented

---

Porting to Your Own Hardware

To adapt this project to a different Ackermann robot with your own chassis, sensors, or actuators, see the Hardware Adaptation Guide for a detailed walkthrough covering URDF modifications, hardware interface rewiring, Nav2 tuning, sensor swaps, and more.