This repository contains executables and tools that allow to easily simulate arbitrary ROS 2 systems and then measures their performance. The system topology can be provided at runtime using JSON files or with command line options.
The framework tracks the following metrics:
- Latency
- Reliability
- CPU usage
- Memory usage
The core of the framework is entirely developed in C++ and it has no external dependencies beside the ROS 2 core libraries. This makes it very easy to be compiled and used on embedded platforms. The iRobot cross-compilation framework can be found at: https://github.com/irobot-ros/ros2-cross-compilation.
Note that this framework is mostly meant for evaluating single process applications. Although it is also able to measure the performance of multi-process applications, not all metrics will be available in this case.
The nodes under test currently don't perform any sort of computation while they are tested. This means that most of the measured resource usage is due to the ROS2 communication.
The only runtime requirement is ROS 2 rolling. The build machine requires Python 3, CMake and colcon.
mkdir -p ~/performance_ws/src
cd ~/performance_ws/src
git clone https://github.com/irobot-ros/ros2-performance
cd ..
colcon build
The irobot_benchmark package contains the main application and example of graph topologies for evaluation.
source ~/performance_ws/install/setup.bash
cd ~/performance_ws/install/irobot_benchmark/lib/irobot_benchmark
./irobot_benchmark topology/sierra_nevada.json
The results will be printed to screen and also saved in the directory ./sierra_nevada_log
.
The irobot_benchmark/topology
directory contains some examples of json files that can be used to create a system.
If you want to create your own, follow the instructions in the performance_test_factory
package:
- performance_test: this package provides the
performance_test::PerformanceNode
class, which provides API for easily adding publishers, subscriptions, clients and services and monitor the performance of the communication. Moreover theperformance_test::System
class allows to start multiple nodes at the same time, while ensuring that they discover each other, and to monitor the performance of the whole system. Moreover, this pacakge contains scripts for visualizing the performance of applications. - performance_test_msgs: this package contains basic interface definitions that are directly used by the
performance_test
package to measure performance. - performance_test_factory: this package provides the
performance_test::TemplateFactory
class that can be used to createperformance_test::PerformanceNode
objects with specific publishers and subscriptions according to some arguments provided at runtime: this can be done through json files or command line options. The interfaces (msg and srv) that can be used in these nodes have to be defined in the so calledperformance_test_factory_plugins
. - performance_test_plugin_cmake: this package provides the CMake function used to generate a factory plugin from interfaces definitions.
- irobot_interfaces_plugin: this package is a
performance_test_factory_plugin
that provides all the interfaces used in the iRobot system topologies. - irobot_benchmark: this package provides our main benchmark application. This executable can load one or multiple json topologies and it creates a ROS2 system running in a specific process from each of them. It also contains the json topologies used for iRobot performance evaluation.
ApexAI provides an alternative valid performance evaluation framework, which allows to test different type of messages. Our implementation is inspired by their work.
- Evaluation of ROS2 Communication Layer
- ROS2 latency and throughput
- Evaluating QoS performances using ROS demos
- Exploring the performance of ROS2
- Time-Sensitive Networking for robotics
- Real-time Linux communications: an evaluation of the Linux communication stack for real-time robotic applications
- Towards a distributed and real-time framework for robots: Evaluation of ROS 2.0 communications for real-time robotic applications
- Time Synchronization in modular collaborative robots
- Reduce delay and jitter in wireless ROS1 networks