planner server died with "invalid transition from state ABORTED with event CANCEL_GOAL"

[zouyonghao]

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04.2 LTS
• ROS2 Version:
  • Foxy
• Version or commit hash:
  • a7955d8
• DDS implementation:
  • no idea, using the default configuration

Steps to reproduce issue

I just set goals several times.

Expected behavior

planner_server should always work.

Actual behavior

planner_server died.

Additional information

See the logs below.

[planner_server-9] [WARN] [1628567027.535320564] [planner_server]: GridBased: failed to create plan with tolerance 0.50.
[planner_server-9] [WARN] [1628567027.535387730] [planner_server]: Planning algorithm GridBased failed to generate a valid path to (-2.87, -2.82)
[planner_server-9] [WARN] [1628567027.535421922] [planner_server_rclcpp_node]: [compute_path_to_pose] [ActionServer] Aborting handle.
[planner_server-9] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[planner_server-9]   what():  goal_handle attempted invalid transition from state ABORTED with event CANCEL_GOAL, at /tmp/binarydeb/ros-foxy-rcl-action-1.1.11/src/rcl_action/goal_handle.c:95
[ERROR] [planner_server-9]: process has died [pid 17164, exit code -6, cmd '/root/dev_ws/install/nav2_planner/lib/nav2_planner/planner_server --ros-args -r __node:=planner_server --params-file /tmp/tmp_p1ktjko -r /tf:=tf -r /tf_static:=tf_static'].

[SteveMacenski]

That is odd. I would be interested if this would happen again or if you found that happened just a single time out of the blue. The error looks pretty low level to me, which I think is not caused by Nav2 itself but a disruption in the ROS2 core stack. The error is from the goal handle which could signal that we're misusing something, but I would want to verify that claim since this is a new error and we have many Foxy users. Getting a traceback would be necessary to debug any further https://navigation.ros.org/tutorials/docs/get_backtrace.html.

Just googling this came up with ros2/rmw_connextdds#8 and ros2/rmw_fastrtps#522 which seems to point at 'not our fault', but I'm not ready to make that claim yet. CC @ruffsl that filed those tickets. Are you using security?

[zouyonghao]

Thanks for your quick reply @SteveMacenski .
This issue just happened once for me yesterday.
It seems we should first make the state to ABORTED and then send a CANCEL_GOAL event to reproduce it, maybe a unit test can do this?
I'll add debug compile options with future tests.

[SteveMacenski]

Sure, I'd definitely appreciate a new unit test 😄. I always love more tests!

Let me know if this happens again + any new information you can give us. If you go some <long time> without seeing it again, I think you could close this as a transient issue in the middleware.

[SteveMacenski]

[bt_navigator-11] [INFO] [1628502832.499278392] [bt_navigator]: Begin navigating from current location to (1.19, -2.35)
[planner_server-9] AddressSanitizer:DEADLYSIGNAL
[planner_server-9] =================================================================
[planner_server-9] ==3623==ERROR: AddressSanitizer: SEGV on unknown address 0x60302b800023 (pc 0x7f01a43a6fb8 bp 0x7f019cf8d110 sp 0x7f019cf8b420 T12)
[planner_server-9] ==3623==The signal is caused by a READ memory access.
[planner_server-9]     #0 0x7f01a43a6fb7 in rmw_fastrtps_shared_cpp::__rmw_trigger_guard_condition(char const*, rmw_guard_condition_t const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x3afb7)
[planner_server-9]     #1 0x7f01a777dca8 in rcl_trigger_guard_condition (/opt/ros/foxy/lib/librcl.so+0x15ca8)
[planner_server-9]     #2 0x7f01a778c7fb  (/opt/ros/foxy/lib/librcl.so+0x247fb)
[planner_server-9]     #3 0x7f01a778b55c  (/opt/ros/foxy/lib/librcl.so+0x2355c)
[planner_server-9]     #4 0x7f01a778c1da in rcl_set_ros_time_override (/opt/ros/foxy/lib/librcl.so+0x241da)
[planner_server-9]     #5 0x7f01a7af510a in rclcpp::TimeSource::set_clock(std::shared_ptr<builtin_interfaces::msg::Time_<std::allocator<void> > >, bool, std::shared_ptr<rclcpp::Clock>) (/opt/ros/foxy/lib/librclcpp.so+0x15c10a)
[planner_server-9]     #6 0x7f01a7af6e98 in rclcpp::TimeSource::clock_cb(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >) (/opt/ros/foxy/lib/librclcpp.so+0x15de98)
[planner_server-9]     #7 0x7f01a7aff367 in std::_Function_handler<void (std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >), std::_Bind<void (rclcpp::TimeSource::* (rclcpp::TimeSource*, std::_Placeholder<1>))(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >&&) (/opt/ros/foxy/lib/librclcpp.so+0x166367)
[planner_server-9]     #8 0x7f01a7b0e6be in rclcpp::AnySubscriptionCallback<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void> >::dispatch(std::shared_ptr<rosgraph_msgs::msg::Clock_<std::allocator<void> > >, rclcpp::MessageInfo const&) (/opt/ros/foxy/lib/librclcpp.so+0x1756be)
[planner_server-9]     #9 0x7f01a7b1232e in rclcpp::Subscription<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void>, rclcpp::message_memory_strategy::MessageMemoryStrategy<rosgraph_msgs::msg::Clock_<std::allocator<void> >, std::allocator<void> > >::handle_message(std::shared_ptr<void>&, rclcpp::MessageInfo const&) (/opt/ros/foxy/lib/librclcpp.so+0x17932e)
[planner_server-9]     #10 0x7f01a7a7002b  (/opt/ros/foxy/lib/librclcpp.so+0xd702b)
[planner_server-9]     #11 0x7f01a7a708ea in rclcpp::Executor::execute_subscription(std::shared_ptr<rclcpp::SubscriptionBase>) (/opt/ros/foxy/lib/librclcpp.so+0xd78ea)
[planner_server-9]     #12 0x7f01a7a710a4 in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) (/opt/ros/foxy/lib/librclcpp.so+0xd80a4)
[planner_server-9]     #13 0x7f01a7a75a4b in rclcpp::executors::SingleThreadedExecutor::spin() (/opt/ros/foxy/lib/librclcpp.so+0xdca4b)
[planner_server-9]     #14 0x7f01a7fa61bc in std::thread::_State_impl<std::thread::_Invoker<std::tuple<nav2_util::NodeThread::NodeThread(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>)::'lambda'()> > >::_M_run() (/root/dev_ws/install/nav2_util/lib/libnav2_util_core.so+0x511bc)
[planner_server-9]     #15 0x7f01a7511de3  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd6de3)
[planner_server-9]     #16 0x7f01a797f608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
[planner_server-9]     #17 0x7f01a71f4292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
[planner_server-9] 
[planner_server-9] AddressSanitizer can not provide additional info.
[planner_server-9] SUMMARY: AddressSanitizer: SEGV (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x3afb7) in rmw_fastrtps_shared_cpp::__rmw_trigger_guard_condition(char const*, rmw_guard_condition_t const*)
[planner_server-9] Thread T12 created by T0 here:
[planner_server-9]     #0 0x482cba in pthread_create (/root/dev_ws/build/nav2_planner/planner_server+0x482cba)
[planner_server-9]     #1 0x7f01a75120a8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd70a8)
[planner_server-9] 
[planner_server-9] ==3623==ABORTING
[ERROR] [planner_server-9]: process has died [pid 3623, exit code 1, cmd '/root/dev_ws/install/nav2_planner/lib/nav2_planner/planner_server --ros-args -r __node:=planner_server --params-file /tmp/tmpgr_7y891 -r /tf:=tf -r /tf_static:=tf_static'].

[SteveMacenski]

[bt_navigator-11] [INFO] [1628499925.752805320] [bt_navigator]: Begin navigating from current location to (0.54, 1.66)
[bt_navigator-11] [ERROR] [1628499925.787389556] [bt_navigator]: Action server failed while executing action callback: "send_goal failed"
[bt_navigator-11] [WARN] [1628499925.787467902] [bt_navigator]: [navigate_to_pose] [ActionServer] Aborting handle.
[bt_navigator-11] [INFO] [1628499939.343112578] [bt_navigator]: Begin navigating from current location to (-2.86, -1.21)
[bt_navigator-11] AddressSanitizer:DEADLYSIGNAL
[bt_navigator-11] =================================================================
[bt_navigator-11] ==14078==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000a8 (pc 0x7f4c699d5fc4 bp 0x616000078980 sp 0x7f4c5c569328 T17)
[bt_navigator-11] ==14078==The signal is caused by a READ memory access.
[bt_navigator-11] ==14078==Hint: address points to the zero page.
[bt_navigator-11]     #0 0x7f4c699d5fc3 in pthread_mutex_lock (/lib/x86_64-linux-gnu/libpthread.so.0+0xbfc3)
[bt_navigator-11]     #1 0x7f4c6a5453fd in nav2_behavior_tree::BtActionNode<nav2_msgs::action::ComputePathToPose>::tick() (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_compute_path_to_pose_action_bt_node.so+0x353fd)
[bt_navigator-11]     #2 0x7f4c69ef5733 in BT::TreeNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6d733)
[bt_navigator-11]     #3 0x7f4c6a1b7e7d in nav2_behavior_tree::RecoveryNode::tick() (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_recovery_node_bt_node.so+0x6e7d)
[bt_navigator-11]     #4 0x7f4c69ef5733 in BT::TreeNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6d733)
[bt_navigator-11]     #5 0x7f4c6a281b59 in nav2_behavior_tree::RateController::tick() (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_rate_controller_bt_node.so+0x5b59)
[bt_navigator-11]     #6 0x7f4c69ef5733 in BT::TreeNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6d733)
[bt_navigator-11]     #7 0x7f4c69ef3602 in BT::DecoratorNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6b602)
[bt_navigator-11]     #8 0x7f4c6a16ab92 in nav2_behavior_tree::PipelineSequence::tick() (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_pipeline_sequence_bt_node.so+0x4b92)
[bt_navigator-11]     #9 0x7f4c69ef5733 in BT::TreeNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6d733)
[bt_navigator-11]     #10 0x7f4c6a1b7e7d in nav2_behavior_tree::RecoveryNode::tick() (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_recovery_node_bt_node.so+0x6e7d)
[bt_navigator-11]     #11 0x7f4c69ef5733 in BT::TreeNode::executeTick() (/opt/ros/foxy/lib/libbehaviortree_cpp_v3.so+0x6d733)
[bt_navigator-11]     #12 0x7f4c6a554bb2 in nav2_behavior_tree::BehaviorTreeEngine::run(BT::Tree*, std::function<void ()>, std::function<bool ()>, std::chrono::duration<long, std::ratio<1l, 1000l> >) (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_behavior_tree.so+0x3bb2)
[bt_navigator-11]     #13 0x7f4c6966d2c2 in nav2_bt_navigator::BtNavigator::navigateToPose() /root/dev_ws/src/navigation2/nav2_bt_navigator/src/bt_navigator.cpp:310:42
[bt_navigator-11]     #14 0x7f4c6977b453 in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:688:14
[bt_navigator-11]     #15 0x7f4c6977b453 in nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::work() /root/dev_ws/install/nav2_util/include/nav2_util/simple_action_server.hpp:144:9
[bt_navigator-11]     #16 0x7f4c6977a9ba in nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()::operator()() const /root/dev_ws/install/nav2_util/include/nav2_util/simple_action_server.hpp:135:68
[bt_navigator-11]     #17 0x7f4c6977a9ba in void std::__invoke_impl<void, nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()>(std::__invoke_other, nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:60:14
[bt_navigator-11]     #18 0x7f4c6977a9ba in std::__invoke_result<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()>::type std::__invoke<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()>(nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:95:14
[bt_navigator-11]     #19 0x7f4c6977a9ba in void std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:244:13
[bt_navigator-11]     #20 0x7f4c6977a9ba in std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:251:11
[bt_navigator-11]     #21 0x7f4c6977a9ba in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/future:1362:6
[bt_navigator-11]     #22 0x7f4c6977a7d9 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:285:9
[bt_navigator-11]     #23 0x7f4c6a53150c in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/root/dev_ws/install/nav2_behavior_tree/lib/libnav2_compute_path_to_pose_action_bt_node.so+0x2150c)
[bt_navigator-11]     #24 0x7f4c699dc47e in __pthread_once_slow (/lib/x86_64-linux-gnu/libpthread.so.0+0x1247e)
[bt_navigator-11]     #25 0x7f4c69779f02 in __gthread_once(int*, void (*)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700:12
[bt_navigator-11]     #26 0x7f4c69779f02 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/mutex:683:17
[bt_navigator-11]     #27 0x7f4c69779f02 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/future:401:2
[bt_navigator-11]     #28 0x7f4c69779f02 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>::_Async_state_impl(std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&)::'lambda'()::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/future:1662:3
[bt_navigator-11]     #29 0x7f4c692bcde3  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd6de3)
[bt_navigator-11]     #30 0x7f4c699d3608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
[bt_navigator-11]     #31 0x7f4c68f9f292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
[bt_navigator-11] 
[bt_navigator-11] AddressSanitizer can not provide additional info.
[bt_navigator-11] SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libpthread.so.0+0xbfc3) in pthread_mutex_lock
[bt_navigator-11] Thread T17 created by T0 here:
[bt_navigator-11]     #0 0x482cba in pthread_create (/root/dev_ws/build/nav2_bt_navigator/bt_navigator+0x482cba)
[bt_navigator-11]     #1 0x7f4c692bd0a8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd70a8)
[bt_navigator-11]     #2 0x7f4c69775aaf in void __gnu_cxx::new_allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >::construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>*, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:147:23
[bt_navigator-11]     #3 0x7f4c69775aaf in void std::allocator_traits<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > >::construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >&, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>*, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:484:8
[bt_navigator-11]     #4 0x7f4c69775aaf in std::_Sp_counted_ptr_inplace<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr_base.h:548:4
[bt_navigator-11]     #5 0x7f4c69775aaf in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>*&, std::_Sp_alloc_shared_tag<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr_base.h:680:6
[bt_navigator-11]     #6 0x7f4c69775aaf in std::__shared_ptr<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::_Sp_alloc_shared_tag<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr_base.h:1344:14
[bt_navigator-11]     #7 0x7f4c69775aaf in std::shared_ptr<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >::shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::_Sp_alloc_shared_tag<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr.h:359:4
[bt_navigator-11]     #8 0x7f4c69775aaf in std::shared_ptr<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > std::allocate_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> >, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > const&, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr.h:701:14
[bt_navigator-11]     #9 0x7f4c69775aaf in std::shared_ptr<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void> > std::make_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >, void>, std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/shared_ptr.h:717:14
[bt_navigator-11]     #10 0x7f4c69775aaf in std::shared_ptr<std::__future_base::_State_baseV2> std::__future_base::_S_make_async_state<std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> > >(std::thread::_Invoker<std::tuple<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/future:1700:14
[bt_navigator-11]     #11 0x7f4c69775aaf in std::future<std::__invoke_result<std::decay<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()>::type>::type> std::async<nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()>(std::launch, nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)::'lambda'()&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/future:1714:18
[bt_navigator-11]     #12 0x7f4c6975e035 in nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::handle_accepted(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >) /root/dev_ws/install/nav2_util/include/nav2_util/simple_action_server.hpp:135:27
[bt_navigator-11]     #13 0x7f4c697822f8 in void std::__invoke_impl<void, void (nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::*&)(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >), nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>*&, std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> > >(std::__invoke_memfun_deref, void (nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>::*&)(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >), nav2_util::SimpleActionServer<nav2_msgs::action::NavigateToPose, rclcpp::Node>*&, std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:73:14
[bt_navigator-11]     #14 0x7f4c697650be in std::function<void (std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >)>::operator()(std::shared_ptr<rclcpp_action::ServerGoalHandle<nav2_msgs::action::NavigateToPose> >) const /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:688:14
[bt_navigator-11]     #15 0x7f4c697650be in rclcpp_action::Server<nav2_msgs::action::NavigateToPose>::call_goal_accepted_callback(std::shared_ptr<rcl_action_goal_handle_t>, std::array<unsigned char, 16ul>, std::shared_ptr<void>) /opt/ros/foxy/include/rclcpp_action/server.hpp:429:5
[bt_navigator-11]     #16 0x7f4c6998db2c in rclcpp_action::ServerBase::execute_goal_request_received() (/opt/ros/foxy/lib/librclcpp_action.so+0x15b2c)
[bt_navigator-11] 
[bt_navigator-11] ==14078==ABORTING

[SteveMacenski]

This trace in the planner server points to it being an issue in the fast rtps rmw. You should file a ticket with them with the information of these 3 tickets and close them on nav2, since I don't even see anything in Nav2 show up in that trace at all, it doesn't even get to that layer before the issue occurs. I'm not an expert in the inner workings of C++ futures, but the BT navigator trace appears to nod to that as well, it looks like its issue is related to the future not being fulfilled due to said crash.

[zouyonghao]

Thanks @SteveMacenski
Now I understand that this ticket #2499 and the #2505 are about the middleware or something else, and this issue seems same as ros2/rmw_fastrtps#522 .
So I opened a new issue ros2/rmw_fastrtps#553 to discuss #2505
But #2504 seems that it is about this repo, maybe we should close this issue and reopen #2504 ?

[SteveMacenski]

I don't understand what you mean, can you explain? I read the BT nav issue being in the futures which are being messed up due to the crash on the promiser side (planner server). That ticket is just a traceback, the issue is described here with the tracebacks, we should not reopen any of those tickets, they contained no actual context or issue to resolve.

[zouyonghao]

Ok, @SteveMacenski
I'll track these issues here.

[SteveMacenski]

Closing this here since it is cross referenced back home to fast-rtps, but I'm open to reopening it if there turns out to be a component to this issue that we can resolve within Nav2. Let me know.