-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using Tinq for ROS 2.0 #7
Comments
Hello, |
The plain C requirement should be ok but there're dependencies on POSIX although things like thread, lock, socket, memory ... creation/handling/management have their own abstraction layers inside the tinq/qeo dds code (with implementations for linux, windows, possibly mac) |
Hey @bramstes thanks for the quick answer. We are really excited about the chances of using Tinq. Although we'd like to make it as deep-embedded as possible we are not so strict about the 200 KB. We probably can get started with a less ambitious aim. Let's say 500 KB. It'll be important to identify which are the POSIX dependencies and how could we bypass them into minimal implementations. It's specially important to make sure that RTPS is working without this dependencies. I'll look forward for your input. Meanwhile i keep diving into your code. |
Guys, I'm currently on holidays with limited access to the internet...(this
|
With the default setup on my 64-bit Linux libdds.so measures up to 4.4Mb unstripped, but only 823Kb stripped. When disabling security it goes down to 3.3Mb and 634Kb respectively. So depending on what you want in or out a goal of 500Kb may be feasable. (Haven't build ARM binaries yet.) |
Hi guys. |
@jvoe @kriver @brunodebus and @bramstes, thanks a lot for your inputs. Let me summarize the conclusions stated so far:
According to this facts, moving forward with Tinq's dds code seems a great solution. My next step will be to create a minimum prototype with FreeRTOS as the underlaying system. I've selected FreeRTOS for it's popularity, weird/relaxed GPL license, and because of its portability. FreeRTOS has ports for many devices and is extremely lightweight. The prototype should include:
|
@vmayoral, |
Hey @jvoe! Thanks a lot for the follow up. With regard the licensing aspect, as you point out, at the OSRF we can't accept a GPL. FreeRTOS offers a relaxed GPL and after reviewing it we consider that it should be sufficient for our users. Regarding the networking stack, you're right. We won't be using the default FreeRTOS+UDP but lwIP which has a nice BSD license and is easily integrated in FreeRTOS. I haven't fully dived yet on lwIP yet but i saw that it has a Anyhow it's really great hearing that the interfaces are hackable :). I'm coding a first prototype at https://github.com/vmayoral/ros2_embedded. Hopefully next week i'll come back with feedback. |
@jvoe @kriver @brunodebus and @bramstes, after the having spent the last week working on freeRTOS (that provides a minimal layer of pseudo-threads called tasks) and a minimal UDP implementation on top of it (coded by us, we discarded lwIP) we are ready to start porting the Tinq's interfaces to our embedded target (STM32F4). Since it's a non-POSIX environment (C99) i've spent some time trying to figure out (besides
A few questions if you allow me:
|
@vmayoral As to your questions:
Let me know how it's going. This is exciting work and I hope you don't encounter too many difficulties. |
@jvoe @kriver @brunodebus and @bramstes, After struggling some time with FreeRTOS and its low level complexities we decided to take a step back and evaluate other RTOS options so we spent the last weeks putting together three prototypes: After having evaluated the three of them we decided to select NuttX for three main reasons:
I'm working now on full speed at https://github.com/vmayoral/ros2_embedded_nuttx which looks quite promising (specially interesting is the NuttX shell NSH which can be hacked to include DDS inspecting capabilities). I ported Is shared memory a big pilar within Tinq's code? Is there an alternative way of interprocesses communication implemented (maybe a slower one?). Would love to hear your opinion in this matter. |
Typically most of the time we used unix domain sockets to do ipc on our gateways but jvoe and Bruno are better placed to answer what is supported in dds (i remember shared memory is something we discussed at some point a.o.in the context of avoiding discovery info dupplication but don't remember us actually implementing it).Since you say the BSD socket interface is supported on nuttx, Unix domain sockets could be an option. But like I said I'd prefer jvoe and/or Bruno to answer :-).CheersBram.-------- Oorspronkelijk bericht --------Onderwerp: Re: [tinq-core] Using Tinq for ROS 2.0 (#7)Van: Víctor Mayoral Vilches Aan: brunodebus/tinq-core Cc: bramstes @jvoe @kriver @brunodebus and @bramstes, After struggling some time with FreeRTOS and its low level complexities we decided to take a step back and evaluate other RTOS options so we spent the last weeks putting together three prototypes: ros2_embedded_nuttx POSIX (pseudo) interface I ported sys and sock so far and I'm working on ipc. Semphores is something we will need indeed in the embedded world however we are not so sure about the shared memory functionality. In general purpose operative systems we understand that several processes sharing memory can lead to a more efficient performance however as we conceive ROS 2.0 for embedded systems, we can't think of an scenario requiring it. Is shared memory a big pilar within Tinq's code? Is there an alternative way of interprocesses communication implemented (maybe a slower one?). Would love to hear your opinion in this matter. —Reply to this email directly or view it on GitHub. |
Hi, IPC is only used to make sure multiple dds instances on the same system get a unique participant id. The partiticipant id is part of the guid (it makes the guid unique on one system) and it is used to choose which ports will be used by this participant for communication. You can compile the code with the define NOIPC to turn of this code (we use this on android where we also cannot use posix ipc). Basically, the code will also work, but not as efficient: it generates a random partiticipant id, and tries to open the corresponding ports. If that fails it will restart and try the next participant id. So you can try with NOIPC first. If you will ever run a lot of participants on the same device ( not the idea I think :-) ) we can figure something else out. |
@brunodebus and @bramstes, Thanks for your answers. Trying it with I'm fighting a bit with the security implemented in DDS. My current configuration is available here. Is there any other flag that you recommend to remove(or add)? Since security is nicely integrated in the code and there's no official support for openssl in NuttX i've decided to omit security to the maximum point for now. I'm hand excluding code like: #include "openssl/ssl.h"
...
#include "dds/dds_security.h" Let me know if you have any comments on this matter please. |
Instead of continuing that path which was a bit senseless because it was breaking all of your work i just droped |
@jvoe @kriver @brunodebus and @bramstes, It seems we are getting way closer 👍. The current prototype with NuttX is providing pretty good results on the integration side. Let me share with you my two current blockers:
I'd say it shouldn't be too hard to code something here that uses
Currently the linker is complaining (i'm trying to build the DDS chat example):
The board we are using has 112KiB + 16 KiB of usable SRAM. @NuttX helped us understand that :
We definitely need to shrink the code size quite a bit if we wish to continue moving forward (specially because we also will need to create a ROS thin layer on top of DDS). Can you please provide suggestions on what we could remove? For now a minimal DDS implementation with RTPS should do it. Once again, we are really happy and thankful for your support. Thanks everyone! |
I removed |
About the memory issue: actually, code size is not the reason for the linking error you are getting, .bss (zero initialized data is). The main culprits for this are the send and receive buffers allocated in dds/src/trans/ip/rtps_ip.c (lines 157 & 172). They are each ~64kB.... (MAX_RX_SIZE/MAX_TX_SIZE) You can change this size (and you will only be able to receive smaller messages) in ./dds/src/include/ri_data.h and your program should be able to link. @jvoe: Does the buffer need to have a specific size/alignment? I cannot remember. |
@brunodebus thanks for the suggestion. That did it. @bramstes, @brunodebus, @kriver and @jvoe: I'm trying now to run the chat on the embedded board but it seems it might need quite a bit of work. Right now, i'm trying to figure out the right configuration for the ParVal_t parameters in NuttX. Usually they are loaded from a file but in this case NuttX has no FS (not that we are using at least). My idea is to set them by default to a reasonable value in the NuttX case. Could you provide some reference on what values these parameters should take? (dds/doc/tdds-env.doc seems empty). Or maybe a set of tips on what values these params should take to be able to run in a embedded system? |
I would suggest that you use the configdata interface to hide the NuttX, of course, does support a variety of file systems. The STM32F4DIS-BB Traditionally, embedded systems keep configuration data in tiny serial FLASH or Hardcoding anything is generally a bad idea. But hiding the source of Greg
|
@vmayoral: for documentation on DDS configuration, as well as for other specifics of Tinq DDS, check the dds/doc/dita/out/user_manual.pdf. |
@jvoe thanks for your answer. I was hoping i could get a link to how things get configured for a sample application. There're many parameters that although well documented, i don't feel familiar enough with the DDS implementation to guess a good value. For example:
It will be great if you could provide a pointer to how things are configured for the |
@vmayoral For the minimal amount of memory on startup, use -DFORCE_MALLOC. This will force all required memory to be malloc-ed from the heap instead of using static memory pools. Your application will only use what it needs then. There will be fragmentation overhead, of course, and this is not as fast as memory pools, but these disadvantages might not be that bad. |
@NuttX and @LorenzMeier, |
The issue with the memory has been fixed. It seems like Keep moving forward. @jvoe will try setting up the DDS Debug shell as soon as possible. |
@bramstes, @brunodebus, @kriver, @NuttX and @jvoe: I finally got the
Any comments will be appreciated. |
I don't understand what the issue is
Looks like a stack overrun. This is very close to overflowing the sp: 10003090 Definitely overran the stack: sp: 10002f48 You will need to increase the stack size. Overrunning the stack can cause all Greg |
@vmayoral So an unmarshalling problem after all. Probably in one of the subfunctions of pid_parse(), called from pid_parse_reader_data() or from pid_parse_writer_data(). Try stepping into that pid_parse() with a first breakpoint in one of the caller functions, examining the parameter_id value each loop iteration until you see it failing. The subfunction that parses that parameter_id can get a breakpoint next to see where it goes wrong. |
@jvoe debugged The The error always comes from this call: Which if I step in corresponds with Particularly it seems that here's where the problem comes in: Not sure how to interpret this matter however in the embedded board i clearly get the messages complaining about an unmarshalling issue. The following picture also shows the package being sent from the Desktop DDS to the Embedded DDS which repeatedly happens (capture available here). |
@vmayoral Looks like the issue is in parsing the typecode of the publication/subscription topic, which isn't the easiest piece of code. Fortunately the Chat topic is not a complex one, so this is doable. Strange that this doesn't work well on the target as the same Chat topic is properly parsed on the desktop with no issues whatsoever. Also, this code was tested on a myriad of other target platforms, Little endian as well as Big endian, including MIPS CPUs, various ARM CPUs (RPi, iOS, lots of different Android phones/tablets), and 32-bit as well as 64-bit x86 architectures. Just a thought, have you tried compiling with normal enums, i.e. 32-bit, as on the desktop? Please try that before continuing the bughunt, since this seems to be the only difference between your toolchain and the ones we use. |
@jvoe i added the flag
My guess is that this happens because the NuttX code does not have this flag enabled thereby they collide. @NuttX any idea on how i can force 32 bit enums within NuttX? |
@vmayoral "any idea on how i can force 32 bit enums within NuttX?" The CFLAGS come from your code. You set them in your Make.defs file |
@vmayoral You might want to run 'make distclean' and ensure you don't have a stale build. |
@LorenzMeier and @NuttX thanks for your input. I |
Do you have the flags also in the board config, e.g. here in both lines? If you use a different board config, then in that board config as well. You also need to reconfigure to bring those changes into the compile / staging area.
|
@LorenzMeier thanks! Adding the flag in the board configuration files did it.
Tried flashing the board with the new firmware and i still get the same unmarshalling problem:
I'll keep looking into it. |
@vmayoral Sorry to hear it didn't help. You seem to be very close to successful communication though. As a drastic workaround, could you try removing -DDDS_TYPECODE from the chat Makefile? Don't know if it will still compile properly, but it effectively switches off the sending/verifying of type information in topics. If it compiles you'll have to do it both on the Desktop and on the board. |
that did it @jvoe! hurray! |
@vmayoral Nice work 👍 Well done! |
@NuttX I'm trying to come up with a simple DDSIMU demo and for that purpose i'm trying to use one of the I2C accelerometers in the board. After having made sure that the sensor is properly recognized (using NSH
Could you advise if there's something wrong with my implementation or the way that i'm using the I2C API? |
@vmayoral I can pretty much guarantee you that there is nothing wrong with the I2C write function. You will need to study the data sheet for the part to assure that you are communicating with it according its specification. Almost certainly you are not. For example, this is probably wrong: result = I2C_WRITE(i2c, ®_val, 1); Is most likely wrong because it will cause a repeated START. Maybe you need: uint8_t data[2]; Read the data sheet! |
@vmayoral By the way, some devices require the repeated start condition. That is why it is impossible to look at code and determine if it is correct or not. The normal way to debug I2C is with a logic analyzer: You have to capture the waveforms that you generate and then compare than with the device's data sheet. The Seleae logic analyzer does a great job with I2C and is highly recommended. |
@vmayoral This probe is really, really good. I recommend to get one: https://www.saleae.com/logic |
@NuttX and @LorenzMeier thanks to both. Will do. |
We've made some tests (including @NuttX free implementation of NuttX shell within the DDS Debug shell) and we got the following results when it comes to RAM requirements:
which incurs in about 631968 Bytes of SRAM. @jvoe does this sound reasonable to you? |
I checked (script) the RAM memory consumption of Tinq running in Desktop (with security and many other functions active) and i
|
In order to verify this matter we played with the parameter and figure out how much extra memory is needed we modified CONFIG_HEAP2_SIZE: 262 144 bytesWith a
131 072 bytesA
65 536 bytes
32 768 bytesSame behaviour, it slowly starts but i can see the DDS packages being sent over the network nicely. 16 384 bytes and belowThere's not enough memory and it's not possible to dynamically instantiate the ChatMsg with xtypes:
Conclusions:Having tested the RAM requirements this way allows to see how the code behaves with different sizes configured. The memory configuration for the microcontroller is defined in the linker script:
With this in mind it seems like:
|
@vmayoral Can you show the output of the 'spool' command? |
@jvoe sure, here it is:
|
@jvoe I'm trying to test interoperability between other DDS implementations and Tinq starting from an IDL message. I've generated the corresponding C code from the IDL message using the idl app however the code generated does not compile (procedure). Do you recall having had problems with this matter? I'll try replicating the code of the chat and build it from there. Also, we are considering creating a middleware for Tinq so that ROS2 can sit on top (something similar to https://github.com/ros2/ros_middleware_opensplice). Although Tinq is highly configurable with the compiling definitions, would it be ok if we create a shared library so that ROS2-Tinq applications can compile against? |
@vmayoral The 'spool' output shows a normal memory usage pattern - nothing abnormal to see there. I'm not sure where the huge amount of data comes from. The 'idl' tool was written as a first attempt to convert pre- X-types IDL data specifications to TSM C-data structs which are an intermediate format used for typecode generation. It sort of works, except that it has a few restrictions:
I often used it myself with a bit of manual 'tweaking' so it is useful. The tool was never finalized/maintained properly since the decision was taken for Qeo to use QDM data definitions (QDM=Qeo Data Model, which is an XML-data representation, based on the DDS XML-type definition from the X-types spec). The QDM supports a subset of the the full X-types specification, whereas using TSMs and/or directly calling X-types functions allow the (almost) complete X-types functionality to be used. The chat app uses extensibility = mutable types, which cannot be handled by the idl tool. It would not be difficult to change it to use 'normal', i.e. pre- X-types type definitions, which can be handled by the idl tool. The reason for this is again a Qeo decision to use exclusively mutable types in order to better handle possible future type changes. As to creating a shared library for ROS2, I don't think the BSD license is against that ... The code is free to use, except for the Qeo copyright. But I'm not a lawyer, so don't take my word for it. ;-) |
@jvoe thanks for the input. I've been trying to actually make an imu_publisher based on the I tried removing the MUTABLE specific code to come back to x-types (which is ideal for us to integrate with ROS2) however the simple
Which fails when fetching the data key. Is this something expected from x-types or is it related to " extensibility = mutable types"? |
Thanks @jvoe that helped progressing. I moved forward with the Tinq (Desktop) <-> OpenSplice (Desktop) interoperability tests and I'm struggling with this matter. It seems that Tinq and OpenSplice can indeed talk but the messages are not passed appropriately. Please refer to ros2/ros2_embedded_nuttx#25 (comment). I have not been able to make it work with RTI's Connext neither (actually not even discovery). Could you please provide input in this matter? Has Tinq been tested against other DDS implementations? |
@jvoe I don't know if you are still looking at the UDP routing logic but if so, I want to give you a heads up that it just got a significant change today. I am working to support multiple network interfaces and that section of code needed some heavy touches to (1) handle the same port numbers on different interfaces and also (2) to properly route incoming UDP packets. The basic change is the with a single network interface, you can mostly ignore the bound IP address of the UDP socket and do everything with the UDP port. But not so with multiple network interfaces. If you are still looking at this, your inputs and thoughts would be appreciated. Greg |
@NuttX I did have a look at the UDP and socket code at the time when we discussed it. When @vmayoral managed to get the receive sockets working I stopped changing the code, figuring it wasn't that useful anymore. There were only a few changes done, tbh, mainly data, i.e. I just added a queue structure to the UDP socket and some bookkeeping variables, and added init and cleanup code for those. So nothing of much use to you, I guess. |
@brunodebus and @mouse256, trying to use this way to reach you (e-mail didn't work so far):
This is Víctor Mayoral Vilches, working at OSRF. Esteve contacted you a while ago showing our interest in Qeo and Tinq. I am the person pushing forward the ROS 2.0 development on the embedded/deep-embedded side and we are considering using Tinq's code for this purpose.
Our goal is to use a minimal DDS implementation in plain C (ideally ANSI C11) that can run in bare-metal (cortex-m3, -m4, -m0, msp430, ...) so we can't assume a POSIX interface.
So far we have the following architecture in mind:
How does this sound to you? Would it be fine if we contributed to tinq-core to make it deep-embedded capable? If so, maybe you could clarify to me the following aspects:
After having spent some time looking at the code I admit that it's a quite amazing piece of work. I think it'll be great to use Tinq as a fundamental piece of software for embedded robotics.
The text was updated successfully, but these errors were encountered: