Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconsider Node to Participant mapping #250

Merged
merged 11 commits into from
Jun 11, 2020
223 changes: 223 additions & 0 deletions articles/151_node_to_participant_mapping.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
---
layout: default
title: Node to Participant mapping
permalink: articles/node_to_participant_mapping.html
abstract: This article analyzes the performance implications of enforcings a one-to-one mapping between ROS nodes and DDS participants, and propose alternative approaches.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
author: '[Ivan Paunovic](https://github.com/ivanpauno)'
published: true
categories: Middleware
---

{:toc}

# {{ page.title }}

<div class="abstract" markdown="1">
{{ page.abstract }}
</div>

Original Author: {{ page.author }}

## Introduction

### What is a `Node`?

In ROS, a `Node` is an entity used to group other entities.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
For example: `Publishers`, `Subscriptions`, `Services`, `Clients`.
`Nodes` ease organization and code reuse, as they can be composed and launched in different ways.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

### What is a `Domain Participant`?

A `Domain Participant` is a type of DDS entity.
`Participants` also group other entities, like `Publishers`, `Subscribers`, `Data Writters`, `Data Readers`, etc.
But participants do more than that:

- Each `Participant` does discovery by its own.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
Creating more than one `Participant` increases cpu usage and network IO load.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
- Each `Participant` keeps track of other `Domain Participants` and DDS entities.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
Using more than one will duplicate that data.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
- Each `Participant` may create multiple threads for event handling, discovery, etc.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
The number of threads created per participant depend on the DDS vendor (e.g.: [connext](https://community.rti.com/best-practices/create-few-domainparticipants-possible)).
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

For those reasons, `Participants` are heavyweight.

### Current status

There is a one-to-one mapping between `Nodes` and `DDS Participants`.
This simplified the design, as `DDS Participants` provide the same organization that a `Node` needs.
The drawback of this approach, is that performance is deteriorated.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
Furthermore, the maximum number of `Domain participants` is rather small.
For example, [RTI connext](https://community.rti.com/kb/what-maximum-number-participants-domain) is limited to 120 participants per domain.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

## Proposal

The goal of this proposal is to improve overall performance by avoiding the creation of one `Domain Participant` per `Node`.
API changes will be avoided, if possible.

### What is a participant mapped to?

There are two main alternatives:
- One participant per process.
- One participant per context.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

The second approach allows more flexibility.
Considering that by default there's only one context per process, it won't lower the performance.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
Moreover, a mechanism for re-using the same participant in two separete contexts could be added.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

### What is a Node now?
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

There's no lightweight DDS equivalent of a ROS `Node`, so these must be implemented on top of it.
A `Node` should be able to:
- Create other entities as `Publishers`, `Subscriptions`, `Services` and `Clients`.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
`Nodes` should own those entities, that is to say, those entity shouldn't outlive a `Node`.
- List all its entities.

For all the entities, it should be possible to get the `Node` that created them.
Each `Participant` should store all the information needed about its nodes, and communicate it other `Participants`.

### How `Node` information is communicated?

#### Using a topic

The name of all the available `Nodes`, and its `Publishers`, `Subscriptions`, `Services`, `Clients` should be available for every `Participant`.
This information can be communicated using a `topic`.
That topic will be an implementation detail and hidden to the user (i.e.: the `rt/` prefix won't be added to this `DDS topic`).

A message could be sent for:
- Each `Node`
- Each `Participant`

The second option reduces the amount of messages.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
It also allow organizing the data using the `Participant` GUID as the key.
It's not possible to organize the data using the `Node` name as a key, because it can collide.
`Node` name uniqueness can be enforced using a collision resolution mechanism, but it can be detected beforehand (i.e.: this information will be needed by the resolution mechanism).
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
In the following, the second option will be considered.

##### State Message

Each `Participant` will send a message representing their state.
A keyed topic could be used for communicating it.
The `Participant` GUID can be used as the key.
This helps for keeping only one message per `Participant` in the history (see `QoS for communicating node information`).
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
The rest of the message will be a sequence of with the information of each node.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
That message should contain the `Node` name, and four sequences:
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
- GUID of its `Publishers`
- GUID of its `Subscriptions`
- GUID of its `Services`
- GUID of its `Clients`

Vector bounds: TBD

##### QoS for communicating node information

Each published message should be available to late `Subscribers`, and only the last message of each key should be kept.
For that reason, the QoS of the `Publishers` should be:

- Durability: Transient Local
- History: Keep Last
- History depth: 1
- Reliability: Reliable

Considering that a keyed topic will be used, in which the history depth apply for each key, only one `Publisher` per process will be needed.
In that case, `unregister_instance` can be used for delete that key from the history (see [RTI Managing Data Instances](https://community.rti.com/static/documentation/connext-dds/5.2.3/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/Content/UsersManual/Managing_Data_Instances__Working_with_Ke.htm)).


The configuration of the `Subscriber` QoS depends on how the data will be accessed later:
- Polled using `Subscriber` read method when needed.
- Listened and organized in a local cache.

The second option allows better organization of this information (e.g.: in hash tables).
In the first case, the QoS of the `Subscriber` should be:

- Durability: Transient Local
- History: Keep Last
- History depth: 1
- Reliability: Reliable

In the second case, durability can be changed to `Volatile`.

#### Using USER_DATA and GROUP_DATA QoSPolicy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be good to collect these in an alternatives section at the end. I think it's cleaner to state what the proposal is, perhaps mentioning that alternatives were considered and they are described in the appendix, and then you can expand on the alternatives and why they were not taken there.


Each `Participant` could have in its user data the list of node names that owns.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
When this data is changed, each `ParticipantListener` will be notified.
This is not a good option, as `UserData` is just a sequence of bytes.
Organizing a complex message in it won't be easy nor performant.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, would it not be possible to reuse the proposed IDL for encoding the ROS discovery info to serialize it into the UserData paload of the discovery packet, so that users wouldn't have seperate discovery layers that act outside the middlewares own QoS/security settings?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's definitely an option.
The only problem, is that that will require the user data to be sent again when modified. Though that's supported in DDS, I don't see support of that in some implementations (particularly, I don't think FastRTPS support that).

There was another way of communicating this information, using userData from the Participant, DataWriter and DataReader, that didn't require them to change (the document mentions using Publisher/Subscriber groupData, but that can be replaced with DataWriter/DataReader userData). Again, the limitation in this case was the lack of support in FastRTPS for DataWriter/DataReader userData (or Publisher/Subscribe GroupData).

The solution that I like the most, is the one combining Participant/DataWriter/DataReader userData, because information doesn't need to be updated and is just provided once. In this case, we could also take advantage of using a ROS message and serializing it there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, the limitation in this case was the lack of support in FastRTPS for DataWriter/DataReader userData (or Publisher/Subscribe GroupData).

Is there an open issue on this we could track, or do we need to open a new one?

The solution that I like the most, is the one combining Participant/DataWriter/DataReader userData, because information doesn't need to be updated and is just provided once.

Just to clarify, for every new DDS DataReader/DataWriter pair that a new node would add to a DDS participant, that pair could broadcast UserData that would include the ROS level discovery info for that respective node? If a node subscribes to the same topic as another node in the same context, is a new DataReader still added to the participant? Similarly for publishing ad DataWriters as well?

I guess if the two nodes have different QoS settings for the same topic, then such might need to be the case. But if the QoS setting are the same, would separate DataReader/DataWriters still be necessary for other reasons like ownership of messages in the message queue history for callbacks.

I'm just trying to figure out if the UserData from DDS DataWriter/DataReader would be unique to a node, or from a collection of nodes that share that DataWriter/DataReader instance by virtue of being in the same DDS participant or ROS context.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an open issue on this we could track, or do we need to open a new one?

It's commented in the code that there's no support for DataWriter/DataReader userData/groupData. You can open a ticket to ask for it to be added.
In the case of Participant userData there's no API allowing to modify it after the Participant gets created. A ticket could be opened too.

Just to clarify, for every new DDS DataReader/DataWriter pair that a new node would add to a DDS participant, that pair could broadcast UserData that would include the ROS level discovery info for that respective node?

Yes.

If a node subscribes to the same topic as another node in the same context, is a new DataReader still added to the participant? Similarly for publishing ad DataWriters as well?

Yes.

I'm just trying to figure out if the UserData from DDS DataWriter/DataReader would be unique to a node, or from a collection of nodes that share that DataWriter/DataReader instance by virtue of being in the same DDS participant or ROS context.

Currently, most implementations are creating a DDS Publisher/Subscriber for each ROS Publisher/Subscription. Some are just creating a DDS Publisher/Subscriber, and creating one DataWriter/DataReader per ROS Publisher/Subscription. The last solution is probably the ideal one.


I will correct I previous mistake I made.
The idea to combining Participant/Endpoint userData doesn't avoid it to be needed to be updated. Particularly, Participant userData would still be needed to be updated:

  • Participant userData would be a list of node names/namespace + context name: Updated when a new node is created/destroyed.
  • DataWriter/DataReader userData is the node name and namespace of the creator. There's no need to resend it.

The advantage of this combination is that less information has to be resent, and "messages" (userData content) is simpler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea to combining Participant/Endpoint userData doesn't avoid it to be needed to be updated. Particularly, Participant userData would still be needed to be updated:

  • Participant userData would be a list of node names/namespace + context name: Updated when a new node is created/destroyed.
  • DataWriter/DataReader userData is the node name and namespace of the creator. There's no need to resend it.

Can all of these out-of-band info be encrypted ? if not which ones can't be?
I'm trying to evaluate if we would end up leaking system information in a similar manner to ros2/sros2#172 (and maybe leaking topic info as well)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can all of these out-of-band info be encrypted ? if not which ones can't be?

I guess DDS build-in topics are all encrypted if you're using security (including userData info), I would be extremely surprised if not.


After reading ros2/sros2#172, it seems that that information is not being encrypted.
I'm not completely sure if that's a bug in fastrtps or the DDS security specification doesn't cover that. But this info could (and should) be encrypted.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a node subscribes to the same topic as another node in the same context, is a new DataReader still added to the participant? Similarly for publishing ad DataWriters as well?

I guess if the two nodes have different QoS settings for the same topic, then such might need to be the case. > But if the QoS setting are the same, would separate DataReader/DataWriters still be necessary for other reasons like ownership of messages in the message queue history for callbacks.

The DDS interface pretty much requires a DataReader for each ROS2 subscription, regardless of QoS settings, because you take the data. And in the current ROS2 model, publishers can't share DataWriters because the publisher GID is accessible to the subscribers. (If it weren't, for obvious reasons it'd still only be the keep-all writers that one could combine.)

Naturally one could multiplex everything on a single reader/writer by re-implementing a ton of things and putting additional information in the generated IDL, &c. — but that would be a bit silly.

I guess DDS build-in topics are all encrypted if you're using security (including userData info), I would be extremely surprised if not.

After reading ros2/sros2#172, it seems that that information is not being encrypted.
I'm not completely sure if that's a bug in fastrtps or the DDS security specification doesn't cover that. But this info could (and should) be encrypted.

From the way I read the specs, participant discovery data must always be sent in the clear because it is what bootstraps the protocol, and this unfortunately includes the “user data” QoS. Reader and writer discovery is encrypted (when required by the governance file) and that also protects the topic/group/user data fields. (Checked it with Cyclone DDS; @ruffsl, @vmayoral, re ros2/sros2#172, I’d be interested in knowing via which path the type information was leaked, I can’t find any type name in a capture.)

The idea to combining Participant/Endpoint userData doesn't avoid it to be needed to be updated. Particularly, Participant userData would still be needed to be updated:

  • Participant userData would be a list of node names/namespace + context name: Updated when a new node is created/destroyed.
  • DataWriter/DataReader userData is the node name and namespace of the creator. There's no need to resend it.

The advantage of this combination is that less information has to be resent, and "messages" (userData content) is simpler.

It seems to me there is an alternative to modifying the participant user data (although it is spec’d feature and pretty widely supported by DDS implementations): simply infer the nodes from the reader/writer info. There are a few downsides I can see to this, but all seem rather minor:

  • Getting a list of nodes directly from DDS discovery info requires iterating over all readers & writers. Maintaining a better index is easy — and the PR on rmw_dds_common already implements that, right?
  • A ROS2 node without readers or writers would not be visible in the network. It won’t participate either, so its being invisible doesn’t seem like a big deal.
  • If a single process/context/DDS participant may have multiple instantiations of the same ROS2 node, merely putting the node names in the reader/writer info is not enough. That can solved by adding a unique node identifier.

You could put this info in the “user data” QoS of each reader/writer, or you could require that no DDS Publisher/Subscriber be used for more than one ROS2 node and put it in the “group data” QoS of the publisher/subscriber. Each reader/writer’d effectively end up distributing a copy of it anyway in the discovery, but I’d say it is slightly more elegant.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naturally one could multiplex everything on a single reader/writer by re-implementing a ton of things and putting additional information in the generated IDL, &c. — but that would be a bit silly.

100% agreed.

From the way I read the specs, participant discovery data must always be sent in the clear because it is what bootstraps the protocol, and this unfortunately includes the “user data” QoS

That's good to know.

I’d be interested in knowing via which path the type information was leaked, I can’t find any type name in a capture.

It was leaked via Participant userData.

Getting a list of nodes directly from DDS discovery info requires iterating over all readers & writers. Maintaining a better index is easy — and the PR on rmw_dds_common already implements that, right?

To clarify, the current PR uses a topic where this data is published.
This is discussing one of the alternatives.

To answer the question: No, the list of nodes would be available from the userData field of all Participants (which would be a list of node names and namespaces).

i.e.:
Participant user data is used to share a list of nodes, from where you can then print a node list.
Reader/Writer userData is used to specify which node created it.

A ROS2 node without readers or writers would not be visible in the network. It won’t participate either, so its being invisible doesn’t seem like a big deal.

No, same as above.

If a single process/context/DDS participant may have multiple instantiations of the same ROS2 node, merely putting the node names in the reader/writer info is not enough. That can solved by adding a unique node identifier.

Not exactly. If each context checks that there's not repeated node name (which can be easily done), then from both the Participant guid and node name you have an uniquely identified node (though adding a unique node identifier sounds pretty reasonable).


Similarly to `UserData`, `GroupData` is a available in `Publishers` and `Subscribers`.
These entities only need to communicate the GUID of the `Participant` and the `Node` name from which it was created.
This idea can be combined with a topic just publishing the list of `Node` names, without including all the other vectors in the message.
Although, it is more difficult to communicate this information for `Services` and `Clients`, as they use behind the scenes just a `DDS Publisher` and `Subscriber`.

### Other implications

#### Security

In `DDS`, security can be specified at a `Participant` level.
If one `Node` is mapped to one `Participant`, individual configuration of its security key and access control policy is possible.
From a security point of view, only being able to configure it at a `Participant` (or per process) level is enough.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
There's not much sense on having different access control policies for `Nodes` in the same process.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
As they share the same address space, other vulnerabilities are possible.

##### How to create a new security key?

Before, we were creating a key for each `Node`.
The full name of the node was used for creating it.

If we create one `Participant` per context, we will only need a key for each of them, and not one per `Node`.
There are two alternatives:

- Add the concept of `Context` name (or `Participant` name).
In this way, the key of each participant could be specified independently.
- Use one key per process.
All the `Participants` within one process will use the same key (is that possible?).

##### How to specify access policies?

Access control policies could still be specified per `Node` basis.
When a `Participant` is created, it should look at the access control policies of each of its `Nodes` and compose them in a single configuration.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

QQ:
- Is it possible to add more access control policies after creating the `Participant` (e.g.: When later creating a `Node`).

#### Node Name Uniqueness
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

In `Dashing` and before, `Node` name uniqueness is not enfornced.

When creating only one `Participant` per `Context`, we can distinguish two cases:
- There is an overlap between the name of two `Nodes` created within the same `Context`.
This case can be trivially solved.
- There is a collision with the `Node` name created from another `Context`.
By the nature of discovery, when a collision is detected, it's not possible to know what `Node` was created first without extra information.
A collision resolution mechanism have to be decided for solving which `Node` continues living.
A `timestamp` of the `Node` creation published in the state message can help to solve the problem.

If we don't change the `Node` to `Participant` mapping, the last item still stands and should be solved in a similar fashion.

#### Ignore local publications option

There's an `ignore_local_publications` option that can be set when [creating a subscription](https://github.com/ros2/rmw/blob/2250b3eee645d90f9e9d6c96d71ce3aada9944f3/rmw/include/rmw/rmw.h#L517).
That option avoids receiving messages from `Publishers` within the same `Node`.
This wasn't implemented in all the rmw implementations (e.g.: [FastRTPS](https://github.com/ros2/rmw_fastrtps/blob/099f9eed9a0f581447405fbd877c6d3b15f1f26e/rmw_fastrtps_cpp/src/rmw_subscription.cpp#L118)).

For emulating this behavior, messages could be ignored by checking from what `Node` the `Publisher` was created.
This should be possible by querying the state messages, or the local chache were they are organized.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved


#### Intra process communication

Currently, intra-process communication can be enable disabled in each `Publisher` and `Subscription`.
There is only one reason for that: intraprocess communication doesn't support all QoS policies.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

Inter process messages from `Publishers` that can also communicate with a `Subscription` using the intra process layer are ignored before handling the callback.
The same problem will happen when having only one `Participant` per context, and it can be solved in the same fashion.

If in the future our intra process communication support all the QoS policies, we could forbid the possibility of enabling and dissabling it at `Node`, `Publisher`, `Subscription` level.
Configuring intra process communication with `Context` granularity should be enough.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved

#### Launching rclpy nodes

In `Dashing` and before, a container for dinamically composing `rclpy Nodes` is not available.
ivanpauno marked this conversation as resolved.
Show resolved Hide resolved
If this is not added, launching multiple `rclpy Nodes` in a launch file will create multiple participants.
That will make the performance worse, compared with composing `rclcpp Nodes`.
A `rclpy` component container should be added to solve the problem.
A generic container can also be considered, allowing to dinamically load `Nodes` from both clients.