ROS 2 Agnocast

An rclcpp-compatible true zero-copy IPC middleware that supports all ROS message types, including message structs already generated by rosidl.

This middleware is based on the following paper and the corresponding prototype. The paper has been accepted to IEEE ISORC 2025 (pdf).

• T. Ishikawa–Aso and S. Kato, “ROS 2 Agnocast: Supporting Unsized Message Types for True Zero-Copy Publish/Subscribe IPC,” in Proc. 28th Int. Symp. Real-Time Distributed Computing (ISORC), 2025, pp. 1–10.

BibTeX

@inproceedings{ishikawa2025ros,
  title={ROS 2 Agnocast: Supporting Unsized Message Types for True Zero-Copy Publish/Subscribe IPC},
  author={Ishikawa--Aso, Takahiro and Kato, Shinpei},
  booktitle={2025 28th International Symposium on Real-Time Distributed Computing (ISORC)},
  pages={01--10},
  year={2025},
  organization={IEEE}
}

Table of Contents

• Table of Contents
• Supported Environments
• For Users
• For Developers
• Debug
• Troubleshooting
• Documents

Supported Environments

Agnocast is currently available in the following environments. This reflects the current status, and support is expected to expand in the future.

Category	Supported Versions / Notes
ROS 2	Humble / Jazzy (only with rclcpp client library)
Linux Distribution	Ubuntu 22.04 (Jammy) / Ubuntu 24.04 (Noble)
Linux Kernel	5.x / 6.x series (detailed version matrix not yet available)

Warning: Agnocast service/client is not officially supported yet and the API may change in the future. Use at your own risk.

---

For Users

Clone the repository

Since ROS packages under src/ such as agnocastlib are not yet distributed from the ROS build farm, a source build is currently required. Therefore, to perform the source build, first check out the specific version as follows:

git clone --branch 2.2.0 https://github.com/tier4/agnocast.git
cd agnocast

System Configuration

Agnocast uses POSIX message queues for inter-process notification. The following system parameters may need adjustment.

msg_max: Maximum number of messages per queue (Required)

Agnocast requires increasing the system limit for the maximum number of messages in a queue.

Temporary setting (Current session only):

sudo sysctl -w fs.mqueue.msg_max=256

Permanent setting:

echo "fs.mqueue.msg_max=256" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

queues_max: Maximum number of message queues system-wide (Optional)

Agnocast creates a message queue for each subscriber, so the total number of message queues grows with the number of topics and subscribers. The system default for queues_max is typically 256, which may not be sufficient for large-scale deployments.

You can check the current limit with:

cat /proc/sys/fs/mqueue/queues_max

If you encounter mq_open failed: No space left on device, you may need to increase this limit.

Temporary setting (Current session only):

sudo sysctl -w fs.mqueue.queues_max=1024

Permanent setting:

echo "fs.mqueue.queues_max=1024" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

msgqueue: Per-user POSIX message queue memory limit (Optional)

Each bridge manager message queue consumes memory proportional to the maximum message size and queue depth. When running many Agnocast processes simultaneously, the per-user POSIX message queue memory limit (RLIMIT_MSGQUEUE) may be exceeded, causing mq_open to fail.

To increase the limit, either add the following line to /etc/security/limits.conf:

* - msgqueue unlimited

Or use prlimit to change the limit for the current shell:

sudo prlimit --pid=$$ --msgqueue=unlimited

You can verify the current limit with:

ulimit -q

Setup

Run the setup script to install dependencies:

bash scripts/setup

Manual installation of agnocast-heaphook and agnocast-kmod

If you prefer to install the packages manually instead of using the setup script:

# Create keyrings directory
sudo install -d -m 0755 /etc/apt/keyrings

# Download and install GPG key
curl -fsSL 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xCFDB1950382092423DF37D3E075CD8B5C91E5ACA' \
  | gpg --dearmor | sudo tee /etc/apt/keyrings/agnocast-ppa.gpg >/dev/null
sudo chmod 0644 /etc/apt/keyrings/agnocast-ppa.gpg

# Create repository configuration (deb822 format)
cat <<EOF | sudo tee /etc/apt/sources.list.d/agnocast.sources
Types: deb
URIs: http://ppa.launchpad.net/t4-system-software/agnocast/ubuntu
Suites: jammy noble
Components: main
Signed-By: /etc/apt/keyrings/agnocast-ppa.gpg
EOF

# Install packages
sudo apt update
sudo apt install agnocast-heaphook-v2.2.0 agnocast-kmod-v2.2.0

Build

Build the project:

colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

Run

Insert kernel module.

sudo modprobe agnocast

You can optionally specify the mempool size per process (default: 8GB). Note that this parameter is experimental and may be removed or changed in future versions:

sudo modprobe agnocast mempool_size_gb=16

See shared_memory.md for details on mempool configuration.

Run sample app (different window for each script). The order does not matter.

bash scripts/sample_application/run_listener
bash scripts/sample_application/run_talker

Stop applications and unload kernel module.

sudo modprobe -r agnocast

Bridge Feature

The Agnocast Bridge enables communication between Agnocast nodes and standard ROS 2 nodes by automatically forwarding messages between shared memory (Agnocast) and DDS (ROS 2).

Bridge Modes

Configuration for the Agnocast bridge manager. The mode can be specified using strings (case-insensitive) or integers.

Mode	Accepted Values	Description
Standard (Default)	standard, 1	One bridge manager per process. Used if the variable is unset or invalid.
Performance	performance, 2	Single global bridge manager.
Off	off, 0	Bridge disabled.

Note:

• Values are case-insensitive (e.g., Standard, OFF, Performance are valid).
• If an unknown value is provided, it falls back to Standard mode with a warning.

Usage

# Standard mode (Default)
export AGNOCAST_BRIDGE_MODE=standard
# OR
export AGNOCAST_BRIDGE_MODE=1

# Performance mode
export AGNOCAST_BRIDGE_MODE=performance
# OR
export AGNOCAST_BRIDGE_MODE=2

# Disable bridge
export AGNOCAST_BRIDGE_MODE=off

Performance Mode Setup

Performance mode requires pre-compiled bridge plugins for each message type used. Generate and build them using the following steps:

1. Generate the plugin package:

# For specific message types
ros2 agnocast generate-bridge-plugins --message-types std_msgs/msg/String geometry_msgs/msg/Pose

# Or for all available message types
ros2 agnocast generate-bridge-plugins --all

2. Build the generated package:

colcon build --packages-select agnocast_bridge_plugins

3. Source and run:

source install/setup.bash
export AGNOCAST_BRIDGE_MODE=performance
# Run your application

For detailed information, see Bridge Documentation.

---

For Developers

Clone the repository

Clone the latest main branch for development:

git clone https://github.com/tier4/agnocast.git
cd agnocast

Setup pre-commit

The following command allows clang-format, markdownlint, and KUNIT Test to be run before each commit.

python3 -m pip install pre-commit
python3 -m pip install --upgrade pre-commit identify
pre-commit install

If you want to disable pre-commit, please run pre-commit uninstall.

Build and insert kmod

Note

If you have already installed agnocast-heaphook or agnocast-kmod via apt (i.e., bash scripts/setup), please remove them first.

Build.

bash scripts/build_all

Check if there is a libagnocast_heaphook.so in install/agnocastlib/lib.

$ ls install/agnocastlib/lib | grep libagnocast_heaphook
libagnocast_heaphook.so

Insert kernel module.

cd agnocast_kmod
sudo insmod agnocast.ko
sudo lsmod

Test

You can build, test and generate the coverage report by following:

bash scripts/test/test_and_create_report

Kernel Module Test

A custom kernel with the following CONFIG enabled is required to run KUnit Test and obtain the coverage report (sample custom kernel is placed here).

• CONFIG_KUNIT=y
• CONFIG_GCOV_KERNEL=y

If booting with the custom kernel, the following script can be used to run unit tests on kernel modules and generate coverage reports.

bash scripts/test/run_kunit

You can also use pre-commit

---

Debug

Check the kernel log.

sudo dmesg -w

To use dynamic_debug for dynamically outputting debug logs, please run the following command as super user:

sudo su
echo 'file agnocast_main.c +p' > /sys/kernel/debug/dynamic_debug/control

Check if dynamic_debug is enabled by running the following command. If the right side of the = is p, it is enabled. (If it's _, it is disabled.)

sudo cat /sys/kernel/debug/dynamic_debug/control | grep "agnocast_main.c"
/.../agnocast/agnocast_kmod/agnocast_main.c:810 [agnocast]release_msgs_to_meet_depth =p "Release oldest message in the publisher_queue (publisher_pid=%d) of the topic (topic_name=%s) with qos_depth %d. (release_msgs_to_meet_depth)\012"
/.../agnocast/agnocast_kmod/agnocast_main.c:367 [agnocast]insert_message_entry =p "Insert an entry (topic_name=%s publisher_pid=%d msg_virtual_address=%lld timestamp=%lld). (insert_message_entry)"

To use dynamic_debug, the Linux kernel configuration must have CONFIG_DYNAMIC_DEBUG set to y. If CONFIG_DYNAMIC_DEBUG is not enabled in your environment, perform a debug build with:

make CFLAGS_agnocast.o="-DDEBUG"

Refer to the Linux kernel documentation on kbuild for more information about compilation flags.

Troubleshooting

Migrating from old PPA setup

If you previously installed Agnocast using the old add-apt-repository method, remove the old configuration before running scripts/setup:

# Remove old repository configuration
sudo add-apt-repository --remove ppa:t4-system-software/agnocast
sudo rm -f /etc/apt/sources.list.d/*agnocast*.list
sudo rm -f /etc/apt/trusted.gpg.d/*agnocast*.gpg

Shared memory and message queue cleanup

Although Agnocast includes cleanup procedures for resources like shared memory and message queues, these resources may sometimes remain in the system. If you notice that available system memory decreases every time you run an Agnocast-enabled application, you'll need to remove leftover shared memory objects by running:

rm /dev/shm/agnocast@*

Additionally, if you encounter the error mq_open failed: No space left on device, it means that the system has reached the maximum number of message queues. In that case, first try removing leftover message queues by running:

rm /dev/mqueue/agnocast@*
rm /dev/mqueue/agnocast_bridge_manager_parent@*
rm /dev/mqueue/agnocast_bridge_manager_daemon@*

If the error persists after cleanup, you may need to increase the system-wide limit on the number of message queues. See the System Configuration section above for how to increase queues_max.

Documents

• shared memory
• message queue
• Autoware integration
• Memory format in heaphook
• Clang-tidy Suppressions
• How to set environment variables
• ros2 command extension
• agnocast::Node and rclcpp::Node interface comparison
• Callback Isolated Executor for Agnocast
• Agnocast-ROS 2 Bridge
• Message Filters Design Document
• Message Filters User Guide
• ApproximateTime Algorithm