[META] Decouple interface generation pipeline from CMake

[hidmic]

Feature request

Feature description

As it stands, the rosidl interface generation pipeline is strongly coupled with CMake, and in particular with ament_cmake machinery. There's no easy way for external projects using different build systems (or build systems' generators) to generate their own interfaces, or provide their own generators and/or type support packages, but to delegate to a CMake project.

Pushing as much logic and data (e.g. expected output from any given script) into reusable libraries and scripts would greatly ease integration with other build systems. Most code generation already takes place in separate Python scripts that could be used as a starting point.

In addition to that, though orthogonal, making type support dynamic loading machinery optional may further simplify the process for users that do not need such functionality.

[hidmic]

I'll be putting together a concrete design proposal for a change in the coming weeks.

Early input on specific use cases that folks think warrant support via pipeline generalization is most welcomed. Perhaps @EricCousineau-TRI, @IanTheEngineer, @jacobperron, @koonpeng, or @gbiggs may have some.

[jacobperron]

Thanks for the ping!

One feature that is missing in the current pipeline is the ability to retroactively generate interfaces for a new language. For example, in order to generate support for rcljava (and perhaps other client libraries like rclrs), we need to compile interface packages (e.g. rcl_interfaces) from source.

Ultimately, it would be nice if a user compiling rosidl_generator_java didn't also have to clone all of the interfaces that they plan to use into the same workspace (i.e. a tool should detect any interface packages installed in an underlay and generate language support for them). Taking that one step further, it would be nice if I could sudo apt install ros-rolling-rosidl-generator-java and magically have Java language support for existing (and future) interface packages that are installed on my system. This step might be significantly more challenging depending on the design, I haven't thought about it too much.

[EricCousineau-TRI]

Thanks for filing this! I think we should be sure to monitor performance at the forefront - I'd assume for primarily Tier I platforms.

The main concern is that having CMake do a whole bunch of subprocess dispatch may slow things down a bit, esp. on Windows (DISCLAIMER: I currently have zero stake in Windows use cases, but just wanna make sure we don't disservice peeps).
At a minimum, we should identify a good benchmark to (try to) track any slowdowns this may incur (shifting complex logic from CMake to Python / whatevs), and ensure we can easily rerun those baselines.

Are there already existing benchmarks like that? (e.g. recording metrics from ament about configuration time in CMake?)

And dumb question: Can we assume that any system building ROS 2 components will have access to Python? (I think Python's a prereq, but just wanna make sure!)

[jacobperron]

Can we assume that any system building ROS 2 components will have access to Python?

Python is currently listed as a dependency in REP 2000, though there are efforts to define a "minimal" C++-only installation (see ros-infrastructure/rep#231). So, I suppose Python is not (or in the future may not be) a strict dependency for a system running ROS. But, I think Python is a reasonable requirement for the purposes of generating and building ROS messages. It would basically require an entire rewrite of the interface generation pipeline if we dropped Python (for better or worse).

[clalancette]

But, I think Python is a reasonable requirement for the purposes of generating and building ROS messages.

Yeah, I agree; requiring Python on the build machine seems fine. We just want to make sure it isn't required on the target (unless you are using rclpy and friends, of course).

[koonpeng]

This is my experience when working on a rosidl_generator_nodejs. A third party rosidl generator would probably need the following:

1. A list of all packages and their messages, services and actions.
2. A way to parse those message definitions.
3. The include dirs and link libraries for each package/message.

The list of all packages can be easily gotten from the ament index, but "2" and "3" are more complicated. The only way I know of to parse the message definitons is using rosidl_parser which requires python. For "3", the only reliable way currently is to use cmake. Since a 3rd party rosidl generator would probably need to generate the interfaces at "install time" or at "build time of the consuming application", that brings in depedency on python and cmake somewhere between run time and build time.

It would also be nice if a 3rd party idl generator can easily make reliable incremental builds, we will need to have a way that allows arbitrary build systems to detect changes and selectively rebuild only the messages that have changed.

---

After further testing, I realized that this may affect more than just build systems. There are multiple places in the ros ecosystem that assumes that a message's library can be found in <lib_prefix><package_name>__* (maybe because there is no way to get the library?), however that is not a valid assumption, it is possble for a "valid" message to not follow that convention.

https://github.com/koonpeng/ros_msg_with_diff_pkg_name This is an example message that exports it's library in <lib_prefix>some_other_package__*. Another message that does not follow the convention is libstatistics_collector/msg/DummyMessage, included as part of the base ros2 installation. It builds fine but when trying to publish this message with ros2 cli, you get the error

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Failed to find library 'ros_msg_test__rosidl_typesupport_fastrtps_c'
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:73'

with this new error message:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:116'

rcutils_reset_error() should be called after error handling to avoid this.
<<<

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:116'

with this new error message:

  'type support not from this implementation, at /tmp/binarydeb/ros-foxy-rmw-fastrtps-cpp-1.2.4/src/publisher.cpp:86'

rcutils_reset_error() should be called after error handling to avoid this.
<<<
Failed to create publisher: type support not from this implementation, at /tmp/binarydeb/ros-foxy-rmw-fastrtps-cpp-1.2.4/src/publisher.cpp:86, at /tmp/binarydeb/ros-foxy-rcl-1.1.10/src/rcl/publisher.c:180

This also fails with rclpy and even rclcpp. I haven't been able to test this with ros2bag but I think it would likely fail as well. Given that none of the ros tools work with this message, maybe it was never intended for such a message to be "valid", in this case I would suggest adding a check in the cmake macro/functions to prevent such message from being built.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/common-messages-github-organization/18028/14

[hidmic]

Thank you all for the feedback. I'm still trying to put all the pieces together (this one's really tough 😅). In what follows, I'll reply while I layout my current train of thought.

---

As I understand it, ROS software is organized in packages. A package contains source code and data files, along with a build system (or build system generator) to turn that into usable artifacts (in the most general sense e.g. an executable binary, a shell script, a dynamic library, a Python module, a Java JAR file, etc.). Packages are the smallest distributable unit. Each package is defined in one place and built in full. Packages are usually built in groups using workspaces. There can only be one version or variation of a package in a workspace. However, workspaces can be overlayed such that a version or variation of a given package in the overlay takes precedence over a different version or variation of the same package in the underlay.

Our rosidl pipeline for interface generation fits right into that model. It is a collection of build system tools for parsing interface definitions, generating source code, building libraries, and installing and exporting these libraries, that the build system of any given package (well, currently any ament_cmake package) can use. Interfaces are always part of a package.

I intend to stick to this organization model.

---

Ultimately, it would be nice if a user compiling rosidl_generator_java didn't also have to clone all of the interfaces that they plan to use into the same workspace

@jacobperron I don't think this feature belongs to the pipeline. Interfaces in core packages that are distributed in binary underlays are already built, and rosidl_generator_java is a build system tool. You wouldn't expect upgrading cmake to trigger a re-build. And even then, you would still need all the sources and an install location for new artifacts, which have to belong to a package anyway.

I think pulling package sources into an overlay is the simplest, cleanest, and most explicit solution. Perhaps what we need here is better automation to generate custom overlays.

Are there already existing benchmarks like that? (e.g. recording metrics from ament about configuration time in CMake?)

@EricCousineau-TRI AFAIK no, there are no configuration-time nor build-time benchmarks.

And dumb question: Can we assume that any system building ROS 2 components will have access to Python?

@EricCousineau-TRI As mentioned above, Python is a prerequisite according to REP-2000 and the current pipeline relies on it significantly. That is not a justification considering that we're trying to decouple this from cmake (also a prerequisite and heavily relied on). However, I personally think that keeping it as a build time dependency and even using it as grounds for pipeline extension (i.e. substituting ament_cmake extensions with Python entrypoints) is in our best interest. It will allow us to reuse existing code with reasonable runtime overhead.

A third party rosidl generator would probably need the following:

1. A list of all packages and their messages, services and actions.
2. A way to parse those message definitions.
3. The include dirs and link libraries for each package/message.

@koonpeng I don't agree. A rosidl generator should only be concerned about generating source code for an interface definition during a package build. It shouldn't need to crawl underlays in any way.

By your description of rosidl_generator_nodejs, I'm guessing that you want to generate nodejs bindings using the output of an specific generator in a entire underlay. That's not a package build system tool. It can be made to work, but you'll probably end up with some bindings that are completely detached from the ROS organization model.

I can think of two solutions that stay within the model. One is to re-build with a new generator. Automating custom overlay generation can definitely help. The other solution is to go full dynamic and generate the bindings on the fly, in runtime, in process memory space. That's assuming you can go without actually building anything, loading dynamic libraries and working with opaque types and binary blobs.

It would also be nice if a 3rd party idl generator can easily make reliable incremental builds

@koonpeng Agreed, but this is entirely up to the build system. Generators must only provide enough information for it to be possible.

There are multiple places in the ros ecosystem that assumes that a message's library can be found in <lib_prefix><package_name>__* (maybe because there is no way to get the library?), however that is not a valid assumption, it is possble for a "valid" message to not follow that convention.

@koonpeng I think this is a related but orthogonal problem. Currently, only cmake has access to upstream artifacts information (e.g. via cmake modules), thus why some downstream packages make assumptions like these. Even if we can circumvent it by enforcing file names, this fundamental limitation still shows itself whenever trying to depend on ROS packages from a non cmake build.

One potential solution would be to have packages use the ament index to register their artifacts. This isn't quite what it was meant for though. Alternatively, packages could export this information using formats that can be consumed by tools (e.g. a json dump) and build systems and build system generators (e.g. cmake modules, pkg-config files, bazel Starlark files, gradle Groovy files, etc.).

[koonpeng]

@jacobperron Thanks for the useful insights!

By your description of rosidl_generator_nodejs, I'm guessing that you want to generate nodejs bindings using the output of an specific generator in a entire underlay. That's not a package build system tool. It can be made to work, but you'll probably end up with some bindings that are completely detached from the ROS organization model.

Yeah, what I am building is not really a rosidl generator, but more of a bindings generator if it makes sense. The nodejs ecosystem is too different that makes it unrealistic to somehow tie the bindings into the ROS ecosystem. So my current solution if to crawl through the packages and generate bindings for each message at build time (of the consuming application), would this be the recommended approach for a third party "rcl" library that is distributed outside the ROS ecosystem?

[hidmic]

So my current solution if to crawl through the packages and generate bindings for each message at build time (of the consuming application), would this be the recommended approach for a third party "rcl" library that is distributed outside the ROS ecosystem?

That last bit is key. Yes, you can definitely do this, but in doing so you're effectively leaving the ROS ecosystem. Another ROS package won't be able to depend on and use that generated code (at least not through standard channels).

I think your use case simply requires ROS packages to export their artifacts in a way you can consume. And if you can afford doing this in Python, you get the interface parser for free.

[jacobperron]

I don't think this feature belongs to the pipeline. Interfaces in core packages that are distributed in binary underlays are already built, and rosidl_generator_java is a build system tool. You wouldn't expect upgrading cmake to trigger a re-build. And even then, you would still need all the sources and an install location for new artifacts, which have to belong to a package anyway.

Sure, but perhaps we don't need to think of it as a re-build. All a generator package needs are the interface definition files (e.g. .msg, .srv, .action, and .idl files; or maybe just the .idl files), which are already installed. With this, language-specific code can be generated and built, basically separate from the package where the interface definition files came from. The question becomes where to install the built artifacts? Maybe it's possible to install them back with the original package, or maybe to a dedicated package with a mangled name (e.g. sensor_msgs__rosidl_generator_java). I admit it's a half-baked idea 🤷‍♂️

I think pulling package sources into an overlay is the simplest, cleanest, and most explicit solution.

This works well for developing rosidl generators. I guess the problem I'd like to see addressed is related to releasing and distributing packages for other languages. AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation. This means many ROS packages that depend on client libraries other than rclcpp and rclpy cannot be released and distributed on ROS infrastructure. E.g. We can't bloom-release rcljava because it requires java bindings for rcl_interfaces (at least, I don't think it's possible at the moment). This problem seems coupled with the design of the rosidl generation pipeline. Perhaps it's out-of-scope for this ticket, but I thought it would be good to bring up so it's kept in mind.

[koonpeng]

AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation.

Yeah, this is the main problem I see as well. Even if we make it possible to release packages for other languages, it wouldn't help third party developers who don't have access to the build farm to include their rosidl generators when building the messages.

[hidmic]

All a generator package needs are the interface definition files (e.g. .msg, .srv, .action, and .idl files; or maybe just the .idl files), which are already installed.

This is not strictly true. Generators are provided with interface definition files plus their dependencies. Packages usually don't but could affect that generated code arbitrarily.

Maybe it's possible to install them back with the original package

That breaks the assumption that packages are the smallest distributable unit. Packages built and/or released would no longer be static, but implicitly depend on the state of your installation. I also wonder how we would handle generators that are already present in the underlay and for which there's generated code already. Would re-building rosidl_generator_cpp in workspace B replace files in underlay A? What if we want to rollback?

or maybe to a dedicated package with a mangled name

Would these dedicated packages have their own package.xml? If so, it'd break all downstream packages that depend on interfaces to be found on a given package. If not, we'd be breaking REP-122 (i.e. dropping the correlation between ROS package name and artifact location). We would have to guard ourselves against name clashes in install spaces too.

I guess the problem I'd like to see addressed is related to releasing and distributing packages for other languages. AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation. This means many ROS packages that depend on client libraries other than rclcpp and rclpy cannot be released and distributed on ROS infrastructure.

This problem seems coupled with the design of the rosidl generation pipeline.

To make sure I understand, is this because third-party generators are not present in our core builds and so core interfaces are not generated for these other languages? That's a very good point I had not thought about.

What if, for each interface package as we know it today, we had an interface-only package, a package for each language, and a "metapackage" using group dependencies and ament index lookups to tie it all up? I know it's a big (and verbose?) change, but it'd give us total flexibility to build and install only what's needed when it's needed.

For instance, a CMake package that depends on geometry_msgs would indirectly depend on the geometry_msgs_files package, the geometry_msgs_cpp package, the geometry_msgs_py package, etc. On find_package(geometry_msgs REQUIRED), it'd use the ament index to find these group dependencies and bring them in (those that offer CMake modules). If we install and export correctly, language prefixes won't show in source code e.g. install C++ headers to <install-prefix>/include/geometry_msgs_cpp/geometry_msgs/..., export <install-prefix>/include/geometry_msgs_cpp.

Incidentally, by making each language-specific interface package target a specific set of build system tools, we can configure them more precisely. It'd definitely help me address a subtle problem we have with interface versioning today: the same interface package version can expose completely different APIs if it was built against different build system tool versions. We can land a breaking change w/o touching the package.

[hidmic]

Tagging the @ros2/team, my last proposal will spark controversy.

[ivanpauno]

Our rosidl pipeline for interface generation fits right into that model. It is a collection of build system tools for parsing interface definitions, generating source code, building libraries, and installing and exporting these libraries, that the build system of any given package (well, currently any ament_cmake package) can use. Interfaces are always part of a package.

I intend to stick to this organization model.

I would actually try to do the opposite, make the tool as less aware of that organization model as possible.
The tool, in it's most basic form, consumes IDL files (ros IDL, dds IDL, whatever IDL ...) and dependencies of that IDL (only in the case of nested messages), and given the IDL files and the dependencies generates code for different languages.

i.e.: I would like to have a tool that allows you to do the following:

ros2 generate_idl IDL_FILE_1 ... IDL_FILE_N --depends-lib-dir LIB_DIR_1 ... LIB_DIRM --depends-include-dir INC_DIR_1 ... INC_DIR_M --depends-idl-dir IDL_DIR_1 ... IDL_DIR_M --language-plugins c cpp python java --output whatever/directory/i/want

and then after having something in its more basic form working we could have sugar that is "ros build system" aware and does what you generally want in most cases, e.g.:

ros2 generate_idl IDL_FILE_1 ... IDL_FILE_N --depends-packages PACKAGE_1 ... PACKAGE_M --language-plugins autodetect --output whatever/directory/i/want

PS: I may be missing something and making the tool not "ros build system" aware is more complicated of what I think.

[ivanpauno]

For instance, a CMake package that depends on geometry_msgs would indirectly depend on the geometry_msgs_files package, the geometry_msgs_cpp package, the geometry_msgs_py package, etc. On find_package(geometry_msgs REQUIRED), it'd use the ament index to find these group dependencies and bring them in (those that offer CMake modules). If we install and export correctly, language prefixes won't show in source code e.g. install C++ headers to /include/geometry_msgs_cpp/geometry_msgs/..., export /include/geometry_msgs_cpp.

IMO, this might be a good idea, so rcljava can then create geometry_msgs_java and bloom that.

[sloretz]

To make sure I understand, is this because third-party generators are not present in our core builds and so core interfaces are not generated for these other languages? That's a very good point I had not thought about.

For instance, a CMake package that depends on geometry_msgs would indirectly depend on the geometry_msgs_files package, the geometry_msgs_cpp package, the geometry_msgs_py package, etc. On find_package(geometry_msgs REQUIRED), it'd use the ament index to find these group dependencies and bring them in (those that offer CMake modules). If we install and export correctly, language prefixes won't show in source code e.g. install C++ headers to /include/geometry_msgs_cpp/geometry_msgs/..., export /include/geometry_msgs_cpp.

Definitely agree with trying to benefit third party generator support while refactoring, and I see the benefit to splitting the message packages into per-language packages with geometry_msgs group depending on the generated packages (assuming it can be done without downstream users needing to make code changes), but it seems like this change could be made without moving the generation pipeline away from CMake, which makes it seem like a different issue to me.

On decoupling from CMake, I think the scope might be smaller (but still not small 🙃) than what's been discussed so far. It might be limited to replacing the ament_extension bits that the core and third-party message generators register with and get discovered and invoked by. I sort of assumed a redesign would lead to something like protoc for ROS where you call a program with the .msg/.srv/.action/.idl files and an option for which language generator to use, and the output is all the generated code.

[hidmic]

I would actually try to do the opposite, make the tool as less aware of that organization model as possible.

I fully agree with that. What I meant by sticking to the current model is that I'd much rather keep rosidl tools as build tools and not do any funky thing with packages like retroactive rebuilds.

Definitely agree with trying to benefit third party generator support while refactoring, and I see the benefit to splitting the message packages into per-language packages with geometry_msgs group depending on the generated packages (assuming it can be done without downstream users needing to make code changes), but it seems like this change could be made without moving the generation pipeline away from CMake, which makes it seem like a different issue to me.

This is outside the scope of this ticket, yes. But while I'm at this, I'm trying to land a design that, to the extent that it is possible, naturally fixes the problems we know we have today and supports the features we want in the future. I have the impression we won't be doing this again any time soon.

On decoupling from CMake, I think the scope might be smaller (but still not small upside_down_face) than what's been discussed so far. It might be limited to replacing the ament_extension bits that the core and third-party message generators register with and get discovered and invoked by. I sort of assumed a redesign would lead to something like protoc for ROS where you call a program with the .msg/.srv/.action/.idl files and an option for which language generator to use, and the output is all the generated code.

It's a bit more complicated than that. There will be something like a compiler (for interface generator and interface typesupport generators), but to achieve an:

easy way for external projects using different build systems (or build systems' generators) to generate their own interfaces, or provide their own generators and/or type support packages

the CMake build logic itself (which is by far the largest non-reusable portion of the pipeline) has to be put elsewhere (and then find its way back into CMake, or Bazel, or X). Anyhow, we'll discuss over a design doc soon.

[sloretz]

the CMake build logic itself

What do you mean by build logic? I thought most of the non-reusable stuff was the ament extension boilerplate, defining the output files from each generator, and invoking the code generators with CMake targets.

[ivanpauno]

I'd much rather keep rosidl tools as build tools and not do any funky thing with packages like retroactive rebuilds.

👍 yes, retroactive rebuilds are a bad idea IMO.

[hidmic]

I thought most of the non-reusable stuff was the ament extension boilerplate, defining the output files from each generator, and invoking the code generators with CMake targets.

@sloretz Yeap! Plus configuring the build. Most packages share quite a bit of logic, but some have their own peculiarities (e.g. rosidl_typesupport_connext_cpp and rosidl_generator_py are probably the most diverse ones).

[koonpeng]

What if, for each interface package as we know it today, we had an interface-only package, a package for each language, and a "metapackage" using group dependencies and ament index lookups to tie it all up? I know it's a big (and verbose?) change, but it'd give us total flexibility to build and install only what's needed when it's needed.

For instance, a CMake package that depends on geometry_msgs would indirectly depend on the geometry_msgs_files package, the geometry_msgs_cpp package, the geometry_msgs_py package, etc. On find_package(geometry_msgs REQUIRED), it'd use the ament index to find these group dependencies and bring them in (those that offer CMake modules). If we install and export correctly, language prefixes won't show in source code e.g. install C++ headers to /include/geometry_msgs_cpp/geometry_msgs/..., export /include/geometry_msgs_cpp.

IMO, this might be a good idea, so rcljava can then create geometry_msgs_java and bloom that.

There is just one problem I see with this, when a thirdparty contributor releases a new message package or when an existing message is updated, geometry_msgs_java won't be automatically updated. Does bloom have a "hook system" that allows rcljava to tell the build farm to build _java packages for each new/updated interface package?

[hidmic]

when a thirdparty contributor releases a new message package or when an existing message is updated, geometry_msgs_java won't be automatically updated

Correct. There wouldn't be any special logic for these packages. That is, no automatic package creation, no automatic package releases. TBH I don't think that would scale (i.e. centralized CI/CD for every possible interface, language, and middleware combination), but I'm open to folks arguing otherwise.

[hidmic]

Alright, ros2/design#310 is up. Get there and bash it till its rough edges are gone!

[hidmic]

Discussion at ros2/design#310 is still on-going, but I think we all agree about the need for a unified code generation CLI. I'll open up a ticket to track the progress of that first milestone.