New chunk available callback for the new C++ and C APIs

[budrus]

Brief feature description

So far the new API allows for "normal" polling access and the usage in combination with a waitset. It shall also support the registration of callbacks similar to the old API

Detailed information

The old API had the receive handler for this see. But instead of a thread per subscriber we should go for a thread in the runtime that handles the callbacks (could even be extended to a configurable thread pool in future)

• callbacks available for modern C++ API

• callbacks available for C API

ToDo

• Write design proposal with an usage example
• enableEvent in WaitSet attachable classes should have a signature so that they can be reused by the ReactAl. Change the signature to:

/// @return invalidation callback used by the WaitSet and ReactAl
cxx::MethodCallback<void, uint64_t> enableEvent(
                     const EventTypeEnum event, 
                     iox::popo::TriggerHandle & triggerHandle);

• Implement design proposal
• Rename ActiveCallSet into Listener
• [ ] Create use case based comparison table between WaitSet and callback proposal -> Extend documentation for 1.0 release and website #482
• Update example and write a user documentation
• Add hint that the Listener does not take ownership of the callback. (In doxygen and user docu)
• Rename EventInfo into NotificationInfo
• Update design document to be in sync with the final implementation
• [ ] Update Waitset and Listener section in overview.md -> Extend documentation for 1.0 release and website #482
• Separate WaitSet State enums from Listener Event enums (STATE_HAS_DATA, EVENT_DATA_RECEIVED)
• hasTriggered/resetTrigger refactoring in user trigger since it is automatically resetted

Minor ToDo

• [ ] active_call_set.hpp maybe merge indexmanager functionality into loffli -> Consolidate index queue and loffli  #680
• Remove doesContainEventVariable from TriggerHandle.
• Remove setEventVariable from ChunkQueuePopper.
• [ ] Consider using the index_queue instead of loffli as active_call_set indexmanager -> Consolidate index queue and loffli  #680
• do not iterate over array in addEvent ask event origin instead if event is already attached

[budrus]

To start the discussion, here the possible approaches I currently see. The problem we have to solve has similarities to the event queue discussion in ROS 2. For me we also have to think about whether we use our waitset for managing the execution of callbacks or if we have an event queue as source of information.

Waitset Approach

• we already have it can can reuse it in our runtime (eat your own dog food)
• a thread in the runtime calls wait in a loop, when it returns the callbacks for all subscribers in the returned condition container are executed
• we do not need to adapt the subscriber port, chunk receiver and chunk queue as we use the existing functionality with attaching a condition variable
• the chunks are collected in the chunk queue as usual (according to the capacity)
• we can use the guard condition to release the waitset (e.g. catching Ctrl-C end triggering the guard condition)
• I guess the waitset needs a mutex as we want to attach and detach callbacks during while the runtime thread is executing the wait() call see also Refactoring of waitset #341
• Maybe the function_ref for the callback can be stored within the subscriber. Then the runtime does not have to do a bookkeeping for them, it only has to attach and detach subscribers to the waitset. How would we trigger the waitset call, via a public method of the subscriber is maybe a bad idea?
• How do we ensure that we do not access a deleted subscriber or deleted callable? The user would be responsible for ensuring that the callable attached to a subscriber lives as long as the subscriber (I think we also have this contract for ara::com. Currently the subscriber would call the error handler if it is destroyed and still attached to a waitset. Things are mybe more complicated when we change this, see Refactoring of waitset #341. References to conditions are returned with the wait() call. This could be used even after a detach call.
• The waitset has some costs for iteration over the conditions container and check which one has triggered.

A) void, void callbacks

• we somehow need the contract that sample(s) are consumed in the callback. Otherwise the waitset will still indicate that there are available chunks and call the callback over and over again.
• This would be more a "Ey, you have work to do" callback
• in the callback several samples could be taken and if all available are taken there will also be no further callback as the condition in the watiset is no more fulfilled
• There is only one way to consume chunks and this is the take() call on user API level

B) callback with chunk header

• this would solve to callback endless loop problem as the runtime takes a sample and provides it with the callback
• For each sample one callback
• Do we have to prevent users from calling take as this would be a race in the chunk receiver? So a special access API for the runtime that allows to take samples and trigger the callback

Queue Approach

• We could save the "iterate over the conditions container" of the waitset if we use a queue to signal to the runtime where a chunk was received. This would be a MPSC queue. We then can wait for events on this queue, pop them and execute the related callback
• We have to do the bookkeeping in the runtime. I.e. which subscriber callbacks currently shall be triggered if event come in
• The events must contain the information for the runtime to identify the affected subscriber. Maybe this must be stored in the ChunkQueue

C) Event Queue

• The chunks are pushed in the "normal" chunk queue of a subscriber and only an indication is pushed to the runtime event queue. Only the information to identify the affected subscriber is needed
• Here we have the same problems like the event queue discussed in the ROS2 executor. As we have two queues with different sizes, we would have an event for every chunk that arrived but chunks could be dropped in the chunk queue when overflowing. Then we have some kind of events without chunks.
• For pub/sub combinations with a lot of overflow and dropping of samples we have some unnecessary load with empty events
• We would have to check the subscriber for available chunks and not call the callback if there are no.
• We don't have the waitset overhead but we have one additional multi-process push/pop for every chunk (for the subscribers with registered callback)
• We could have both options, void, void callbacks or callbacks with chunk header

D) Work queue

• The chunks are not pushed in the "normal" queue but directly in a work queue from the runtime. It includes the shared chunk and the indication to which subscriber it belongs
• This would be a big change for all the building blocks. A chunk distributor has to deal with "normal" chunk queues and this special one. But currently has only one interface and associated type
• The bookkeeping that is done in the chunk receiver has to be done by the runtime
• Big difference between callback based subscriber and normal subscriber. I guess you cannot change it during runtime like set/unset callback (could be a problem for ara::com)
• Every chunk is forwarded, there is no more automatic dropping with a safely overflowing queue (should be OK, this is another kind of consume)

[budrus]

Ready for voting? @elfenpiff @FerdinandSpitzschnueffler @MatthiasKillat @dkroenke @elBoberido @ithier @marthtz @mossmaurice

[mossmaurice]

Voting for B)

Main reasons:

• Eat your own dogfood, I think using our own WaitSetmake sense and we quickly spot performance bottlenecks
• It's reusing our current classes (LEGO block approach) and does not introduce new patterns or classes (consistent with what we've done with the building blocks)
• I agree with you, that in case of going for B) we need a special e.g. callbackTake() for the runtime
• If we would go for a queue based approach: How would we statically dimension the queue? What would happen in the overflow case?