ASAN reported SEGV in fastrtps

[zouyonghao]

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04.2 LTS
• Installation type:
  • binaries
• Version or commit hash:
  • Foxy
• DDS implementation:
  • fastrtps
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

Use Nav2 to navigate turtlebot3 in gazebo.

We see crashes when we test Nav2, and it seems some errors are caused by the middleware.

[bt_navigator-11] [INFO] [1628502832.499278392] [bt_navigator]: Begin navigating from current location to (1.19, -2.35)
[planner_server-9] AddressSanitizer:DEADLYSIGNAL
[planner_server-9] =================================================================
[planner_server-9] ==3623==ERROR: AddressSanitizer: SEGV on unknown address 0x60302b800023 (pc 0x7f01a43a6fb8 bp 0x7f019cf8d110 sp 0x7f019cf8b420 T12)
[planner_server-9] ==3623==The signal is caused by a READ memory access.
[planner_server-9]     #0 0x7f01a43a6fb7 in rmw_fastrtps_shared_cpp::__rmw_trigger_guard_condition(char const*, rmw_guard_condition_t const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x3afb7)
[planner_server-9]     #1 0x7f01a777dca8 in rcl_trigger_guard_condition (/opt/ros/foxy/lib/librcl.so+0x15ca8)
[planner_server-9]     #2 0x7f01a778c7fb  (/opt/ros/foxy/lib/librcl.so+0x247fb)
[planner_server-9]     #3 0x7f01a778b55c  (/opt/ros/foxy/lib/librcl.so+0x2355c)
[planner_server-9]     #4 0x7f01a778c1da in rcl_set_ros_time_override (/opt/ros/foxy/lib/librcl.so+0x241da)
[planner_server-9]     #5 0x7f01a7af510a in rclcpp::TimeSource::set_clock(std::shared_ptr<builtin_interfaces::msg::Time_<std::allocator<void> > >, bool, std::shared_ptr<rclcpp::Clock>) (/opt/ros/foxy/lib/librclcpp.so+0x15c10a)
[planner_server-9]     #6 0x7f01a7af6e98 in rclcpp::TimeSource::clock_cb(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >) (/opt/ros/foxy/lib/librclcpp.so+0x15de98)
[planner_server-9]     #7 0x7f01a7aff367 in std::_Function_handler<void (std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >), std::_Bind<void (rclcpp::TimeSource::* (rclcpp::TimeSource*, std::_Placeholder<1>))(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >&&) (/opt/ros/foxy/lib/librclcpp.so+0x166367)
[planner_server-9]     #8 0x7f01a7b0e6be in rclcpp::AnySubscriptionCallback<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void> >::dispatch(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >, rclcpp::MessageInfo const&) (/opt/ros/foxy/lib/librclcpp.so+0x1756be)
[planner_server-9]     #9 0x7f01a7b1232e in rclcpp::Subscription<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void>, rclcpp::message_memory_strategy::MessageMemoryStrategy<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void> > >::handle_message(std::shared_ptr<void>&, rclcpp::MessageInfo const&) (/opt/ros/foxy/lib/librclcpp.so+0x17932e)
[planner_server-9]     #10 0x7f01a7a7002b  (/opt/ros/foxy/lib/librclcpp.so+0xd702b)
[planner_server-9]     #11 0x7f01a7a708ea in rclcpp::Executor::execute_subscription(std::shared_ptr<rclcpp::SubscriptionBase>) (/opt/ros/foxy/lib/librclcpp.so+0xd78ea)
[planner_server-9]     #12 0x7f01a7a710a4 in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) (/opt/ros/foxy/lib/librclcpp.so+0xd80a4)
[planner_server-9]     #13 0x7f01a7a75a4b in rclcpp::executors::SingleThreadedExecutor::spin() (/opt/ros/foxy/lib/librclcpp.so+0xdca4b)
[planner_server-9]     #14 0x7f01a7fa61bc in std::thread::_State_impl<std::thread::_Invoker<std::tuple<nav2_util::NodeThread::NodeThread(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>)::'lambda'()> > >::_M_run() (/root/dev_ws/install/nav2_util/lib/libnav2_util_core.so+0x511bc)
[planner_server-9]     #15 0x7f01a7511de3  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd6de3)
[planner_server-9]     #16 0x7f01a797f608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
[planner_server-9]     #17 0x7f01a71f4292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
[planner_server-9] 
[planner_server-9] AddressSanitizer can not provide additional info.
[planner_server-9] SUMMARY: AddressSanitizer: SEGV (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x3afb7) in rmw_fastrtps_shared_cpp::__rmw_trigger_guard_condition(char const*, rmw_guard_condition_t const*)
[planner_server-9] Thread T12 created by T0 here:
[planner_server-9]     #0 0x482cba in pthread_create (/root/dev_ws/build/nav2_planner/planner_server+0x482cba)
[planner_server-9]     #1 0x7f01a75120a8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd70a8)
[planner_server-9] 
[planner_server-9] ==3623==ABORTING
[ERROR] [planner_server-9]: process has died [pid 3623, exit code 1, cmd '/root/dev_ws/install/nav2_planner/lib/nav2_planner/planner_server --ros-args -r __node:=planner_server --params-file /tmp/tmpgr_7y891 -r /tf:=tf -r /tf_static:=tf_static'].

Related to ros-navigation/navigation2#2499 ros-navigation/navigation2#2505

[fujitatomoya]

@zouyonghao thanks for registering issue.

Could you provide the reproducible procedure about 'test Nav2'? sorry i do not know very much about that, but I would like to try with mainline.

[zouyonghao]

Hi @fujitatomoya
I just use Nav2 with the tutorial here , and this issue only happened one or two times and can not be reproduced stably.
Maybe it's a data race related issue in librcl according to the asan trace?