ROS2 sys_status plugin services callbacks not executing
#1588
Comments
blakermchale
changed the title
sys_status plugin services not being calledsys_status plugin services callbacks not executing
|
I seen that service call problems, but don't really know what's wrong. It could be very random, any plugin can become silent. Note that in mavros v2 each plugin has a separate node instance, so on graph you'll see lots of nodes like Diagnostics exists, just command should be like |
|
I added diagnostics and checkid to my initial post. I also noticed that checkid does not support namespacing outside of the default. The topics for |
|
Try to use that option: mavros/mavros/mavros/cmd/__init__.py Lines 76 to 80 in f1dad6c Note that it should be passed to Since services build on top of DDS topics, i assume that we would have problems with both subscribers and service servers. I saw problems even with |
|
Thanks for that info. I actually found that passing a namespace to Even with using the I've been working in Foxy mostly but tried Galactic today and it did not seem to change the results. I still couldn't get the service callback to work. |
|
Ah, yes, that's because: mavros/mavros/src/mavros_node.cpp Line 50 in f1dad6c But with multi-drone scenario you don't have to use mavros_node. You need to start one Router (read ros2 composite nodes tutorial), and instruct it to connect to drones and uas. Then you can run 2 (or more) UAS with different names (namespaces). Please note that you shuld use different system ids on each system, otherwise you may get wide list of problems. |
|
Hi! @vooon @blakermchale . I'm facing the same issue on ROS2 Galactic. Have you made any progress on this? |
|
@orchydea I haven't found any solution yet and have primarily used RTPS instead as a replacement to mavros |
|
That's a shame @blakermchale. I will try using RTPS or Thanks! |
|
+1 I'm having this issue on ROS2 Foxy as well. Might have to figure out the |
|
@IanBurwell I ended up porting everything to ROS1 (Noetic). I would recommend to do so if your project is at an early stage. It's a pity that this does not work on ROS2 😢 |
|
I've possibly found the workaround of just sending raw MavLink commands, such as the following to force RTL. This is for sure a workaround but I am committed to using ROS2. Worst comes to worst I set up ros2 service call /mavros/cmd/command mavros_msgs/srv/CommandLong "{broadcast: false, command: 176, confirmation: 0, param1: 157, param2: 4, param3: 5, param4: 0.0, param5: 0.0, param6: 0.0, param7: 0.0}" |
|
I'm facing the same problem, Ros2 Foxy. |
|
Nope. But i'm thinking to try use separate executor for each plugin. |
|
I have an issue which I think is related to this one If it's unrelated I can open a separate issue. More details of the command output: |
Did you have any success doing this? Could this be achieved by converting the sys status plugin into its own node and adding it to its own executor? |
|
@Zaaler sorry, i hadn't had time to try that yet. No, it does not have any sense to strip out any plugin. By design they must be loaded to UAS node, which provides some shared state and other abstractions. |
|
I'm having the same problem. FROM ros:foxy-ros-core-focal
RUN apt-get update && apt-get install -qy --no-install-recommends \
python3-colcon-common-extensions \
build-essential gcc git \
iputils-ping \
geographiclib-tools \
python3-vcstool python3-rosinstall-generator python3-osrf-pycommon python3-rosdep
SHELL ["/bin/bash", "-c"]
RUN mkdir -p /ros2_ws/src
WORKDIR /ros2_ws
RUN rosdep init && rosdep update
# install mavros from source
RUN git config --global advice.detachedHead false
RUN rosinstall_generator --format repos mavlink | tee /tmp/mavlink.repos && \
rosinstall_generator --format repos --upstream-development mavros | tee -a /tmp/mavros.repos && \
vcs import src < /tmp/mavlink.repos && \
vcs import src < /tmp/mavros.repos && \
rosdep install --from-paths src --ignore-src -y && \
./src/mavros/mavros/scripts/install_geographiclib_datasets.sh
RUN /bin/bash -c "source /opt/ros/foxy/setup.bash && colcon build"
# ADD ./src /ros2_ws/src
RUN /bin/bash -c "source /opt/ros/foxy/setup.bash && colcon build"
ADD ./params.yaml .
ENV ROS_DOMAIN_ID=42
CMD /bin/bash -c "source install/setup.bash && ros2 run mavros mavros_node --ros-args -r __ns:=/copter --params-file params.yaml"
/**:
ros__parameters:
fcu_url: udp://copter:14551@iris:14555Simulated vehicle is running ArduCopter-4.1 SITL in a separate Subscribing to Are there any configurations for which this does work? Or is this just broken for now? |
|
I've been debugging. If you comment out any callback that makes a client to the cmd/command service name, the system works fine. Those two callbacks are the autopilot_version_cb and the set_message_interval_cb. I plan on continuing to debug it hopefully today. I would assume any plugin that creates a client with cmd/command will cause that plugin to start to hang. From a grep, I found that includes the home_position, sys_status, and mount_control plugins in cpp. Maybe @vooon has some insight from this information. It does sound like those plugins are failing to spin which is causing the client to never receive the response. From placing a print in the cmd/command service, I know that service is processing the request and attempting to send back a response. Just seems like the client end isn't continuing to try and receive it. Any thoughts? Also, where does this issue's importance currently rank in the ROS2 development of MAVROS? Thanks |
|
Maybe i messed up with spinning futures? Unsure. Also in most recent versions of ROS2 it's possible to write async service server, which may be helpful for services like message_interval, which clearly translates to command_long. That bug is a show stopper. So yes, it's important :) |
|
@Zaaler i'm not sure what i've changed, but now i can receive topics (status, diag, imu), call some of services (e.g. params pull). |
|
@vooon I see that the mavros node can be brought up and some of the services can be used now. However, if I bring up the node, and call the set_message_interval service, it hangs still. Once the hanging occurs, all other services can no longer be called like set_mode. Situation seems to persist. Are the times that you have trouble using /mavros/set_mode after you tried to call a service that doesn't return? |
|
Yes, i tried to use I'm still unsure how properly use futures because rclcpp2::spin_future...() throws error saying that the node is already spinning (by UAS::exec, yes). |
|
I found this issue after trying to use the set_mode service and seeing no service responds. After reading @Zaaler comments and @voon responses, I checked the sys_status.cpp plugin code (ros2 version 2.0.3) and found that in set_message_interval_cb() and in autopilot_version_cb() an asynchronous service request is done to "cmd/command" service and then the code waits for the future response, I talking about code like this: (lines 1083- 1085 on sys_status.cpp) I think there is problem with the future.get() : if the requested service run on the same thread as the requester, then this will never happen because the node that needs to handle the request did not received yet the request, causing the thread that runs both services to get stuck, for more details see: Synchronous vs. asynchronous service clients — ROS 2 Documentation: Foxy documentation: [https://docs.ros.org/en/foxy/How-To-Guides/Sync-Vs-Async.html#id3] This can be checked by adding a future status check with timeout regarding the future status, the code after the change: I added such a code to both methods: set_message_interval_cb() and in autopilot_version_cb(), I found that the problem was due to autopilot_version_cb() that is called from a periodic timer, it get stuck and then the SystemStatusPlugin “sys” plugin services will not respond because the thread running the node’s spinner is stuck. The changes I did to the code are not a full solution to the problem, just a way to avoid this plugin to get stuck. |
|
@Michel1968 current ros2 branch has group set for command plugin. But all plugins share same multithreaded runner. Also each plugin is separate node, so i'm not quire sure that we can use sg of one node for clients from another one. Wait_for looks more like a hack, as i suppose it still blocks. But definitely better than complete hung. |
|
#1588 (comment) Signed-off-by: Vladimir Ermakov <vooon341@gmail.com>
|
After playing around with this, I found I was able to reliably prevent the " I wouldn't consider this entirely solved as issue ros2/rclcpp#1707 suggests that calling a service from within a service callback isn't intended. In the future, PR ros2/rclcpp#1709 may be of interest (now in rolling) As for the "hack" it's probably good practice to timeout on service futures anyway as there are still ongoing efforts to improve service reliability especially around discovery at the RMW level (See ros2/rmw_fastrtps#392 & ros2/rmw_cyclonedds#191) |
Issue details
I've built
mavrosin ROS Foxy and am running it usingmavros_node. Services like~/armingwork for thecommandplugin. However, for thesys_statusplugin I cannot call any of the services likeset_modeorvehicle_info_get. I am using the command:ros2 service call /mavros/set_mode mavros_msgs/srv/SetMode "{'custom_mode': 'AUTO.TAKEOFF'}"This service call ends up hanging and never returning. It just says this message:
requester: making request: mavros_msgs.srv.SetMode_Request(base_mode=0, custom_mode='AUTO.TAKEOFF')I've added in rclcpp log info messages to the service callbacks and determined they are never being entered since the messages are not logged.
MAVROS version and platform
Mavros: 2.0.3
ROS: Foxy
Ubuntu: 20.04
Autopilot type and version
[ ] ArduPilot
[X] PX4
Version: master or 9d3e18dbe10700a13f3048f2807dc4b1a62eee62
Node logs
Diagnostics
Check ID
The text was updated successfully, but these errors were encountered: