Skip to content

Fix flaky dwb_critics test ObstacleFootprint.LineCost - maybe related to AddressSanitizer: heap-buffer-overflow READ #4884

New issue
New issue
Closed
Closed
Fix flaky dwb_critics test ObstacleFootprint.LineCost - maybe related to AddressSanitizer: heap-buffer-overflow READ#4884
Labels
bugSomething isn't working
@ewak

Description

@ewak
ewak
opened on Jan 29, 2025

Bug report

Required Info:

  • Operating System:
    • Ubuntu 24.04.1 LTS
  • Computer:
    • 13th Gen Intel(R) Core(TM) i7-1365U - laptop
  • ROS2 Version:
    • rolling
  • Version or commit hash:
  • DDS implementation:
    • RMW_IMPLEMENTATION=rmw_cyclonedds_cpp

Steps to reproduce issue

build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000

/root/nav2_ws/src/navigation2/nav2_dwb_controller/dwb_critics/test/obstacle_footprint_test.cpp:251: Failure                                                                                 
Expected equality of these values:                                                            
  critic->lineCost(3, 3, 0, 50)                                                               
    Which is: 99                                                                              
  50                                                                                          

Trace/breakpoint trap (core dumped)

Thrash it a few times and it breaks in a flaky way.

build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which'

root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 2) . . .
    Which is: 111
root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 11) . . .
    Which is: 220
root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 7) . . .
    Which is: 116
root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 6) . . .
    Which is: 53
root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 5) . . .
    Which is: 105
root@dev-lt-wakem:~/nav2_ws# build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000 2>&1  | egrep 'iteration|Which' | tail -2
Repeating all tests (iteration 6) . . .
    Which is: 116

Patch various projects to enable asan on key components.

diff --git a/nav2_costmap_2d/CMakeLists.txt b/nav2_costmap_2d/CMakeLists.txt
index fc8b8071..d3645d7c 100644
--- a/nav2_costmap_2d/CMakeLists.txt
+++ b/nav2_costmap_2d/CMakeLists.txt
@@ -59,6 +59,12 @@ target_link_libraries(nav2_costmap_2d_core PUBLIC
   tf2_sensor_msgs::tf2_sensor_msgs
 )
 
+#target_compile_options(nav2_costmap_2d_core PUBLIC
+#    -fsanitize=address -fsanitize-recover=address)
+#target_link_options(nav2_costmap_2d_core PUBLIC
+#    -fsanitize=address -fsanitize-recover=address)
+#export ASAN_OPTIONS=new_delete_type_mismatch=0
+
 add_library(layers SHARED
   plugins/inflation_layer.cpp
   plugins/static_layer.cpp
diff --git a/nav2_dwb_controller/dwb_core/CMakeLists.txt b/nav2_dwb_controller/dwb_core/CMakeLists.txt
index e689b318..27c15a9a 100644
--- a/nav2_dwb_controller/dwb_core/CMakeLists.txt
+++ b/nav2_dwb_controller/dwb_core/CMakeLists.txt
@@ -51,6 +51,12 @@ target_link_libraries(dwb_core PUBLIC
   ${visualization_msgs_TARGETS}
 )
 
+target_compile_options(${PROJECT_NAME} PUBLIC
+    -fsanitize=address -fsanitize-recover=address)
+target_link_options(${PROJECT_NAME} PUBLIC
+    -fsanitize=address -fsanitize-recover=address)
+#export ASAN_OPTIONS=new_delete_type_mismatch=0
+
 install(TARGETS dwb_core
   EXPORT dwb_core
   ARCHIVE DESTINATION lib
diff --git a/nav2_dwb_controller/dwb_critics/CMakeLists.txt b/nav2_dwb_controller/dwb_critics/CMakeLists.txt
index 17833b75..acb96017 100644
--- a/nav2_dwb_controller/dwb_critics/CMakeLists.txt
+++ b/nav2_dwb_controller/dwb_critics/CMakeLists.txt
@@ -51,6 +51,12 @@ target_link_libraries(${PROJECT_NAME} PRIVATE
   pluginlib::pluginlib
 )
 
+target_compile_options(${PROJECT_NAME} PUBLIC
+    -fsanitize=address -fsanitize-recover=address)
+target_link_options(${PROJECT_NAME} PUBLIC
+    -fsanitize=address -fsanitize-recover=address)
+#export ASAN_OPTIONS=new_delete_type_mismatch=0
+
 install(TARGETS ${PROJECT_NAME}
   EXPORT ${PROJECT_NAME}
   ARCHIVE DESTINATION lib

Build those components with RelWithDbgInfo

colcon build --packages-select nav2_costmap_2d dwb_critics dwb_core --cmake-args -DCMAKE_BUILD_TYPE=RelWithDbgInfo

Run the test under gdb and have it break on failure

export ASAN_OPTIONS=new_delete_type_mismatch=0
gdb -ex run --args build/dwb_critics/test/obstacle_footprint_tests --gtest_break_on_failure --gtest_repeat=1000

Snippet of failing test.

[ RUN      ] ObstacleFootprint.LineCost
[New Thread 0x7fffeab686c0 (LWP 45369)]
[New Thread 0x7fffea3676c0 (LWP 45370)]
[New Thread 0x7fffe905d6c0 (LWP 45371)]
[New Thread 0x7fffe7d536c0 (LWP 45372)]
[New Thread 0x7fffe67406c0 (LWP 45373)]
[New Thread 0x7fffe3f276c0 (LWP 45374)]
[New Thread 0x7fffe2c1d6c0 (LWP 45375)]
[INFO] [1738187042.526763894] [ns.test_global_costmap.test_global_costmap]:
        test_global_costmap lifecycle node launched.
        Waiting on external lifecycle transitions to activate
        See https://design.ros2.org/articles/node_lifecycle.html for more information.
[INFO] [1738187042.528245626] [ns.test_global_costmap.test_global_costmap]: Creating Costmap
[INFO] [1738187042.529224158] [ns.test_global_costmap.test_global_costmap]: Configuring
[New Thread 0x7fffdf4f66c0 (LWP 45376)]
[INFO] [1738187042.533402628] [ns.test_global_costmap.test_global_costmap]: Using plugin "static_layer"
[INFO] [1738187042.534462909] [ns.test_global_costmap.test_global_costmap]: Subscribing to the map topic (/ns/map) with transient local durability
[INFO] [1738187042.535065598] [ns.test_global_costmap.test_global_costmap]: Initialized plugin "static_layer"
[INFO] [1738187042.535087366] [ns.test_global_costmap.test_global_costmap]: Using plugin "obstacle_layer"
[INFO] [1738187042.535614603] [ns.test_global_costmap.test_global_costmap]: Subscribed to Topics:
[INFO] [1738187042.535654117] [ns.test_global_costmap.test_global_costmap]: Initialized plugin "obstacle_layer"
[INFO] [1738187042.535664649] [ns.test_global_costmap.test_global_costmap]: Using plugin "inflation_layer"
[INFO] [1738187042.536353073] [ns.test_global_costmap.test_global_costmap]: Initialized plugin "inflation_layer"
[New Thread 0x7fffde1ec6c0 (LWP 45377)]
=================================================================
==45344==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x51e000008247 at pc 0x7ffff668b6e6 bp 0x7fffffffa730 sp 0x7fffffffa720
READ of size 1 at 0x51e000008247 thread T0
    #0 0x7ffff668b6e5 in nav2_costmap_2d::Costmap2D::getCost(unsigned int, unsigned int) const /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d.cpp:266
    #1 0x7ffff78b64c7 in dwb_critics::ObstacleFootprintCritic::pointCost(int, int) (/root/nav2_ws/install/dwb_critics/lib/libdwb_critics.so+0x224c7) (BuildId: fd51ce92a44793405ea3dbf2ceb280fbd6c371b9)
    #2 0x7ffff78b65af in dwb_critics::ObstacleFootprintCritic::lineCost(int, int, int, int) (/root/nav2_ws/install/dwb_critics/lib/libdwb_critics.so+0x225af) (BuildId: fd51ce92a44793405ea3dbf2ceb280fbd6c371b9)
    #3 0x5555555f8e67 in OpenObstacleFootprintCritic::lineCost(int, int, int, int) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xa4e67) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #4 0x5555555f2ab9 in ObstacleFootprint_LineCost_Test::TestBody() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0x9eab9) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #5 0x55555564e8bc in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xfa8bc) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #6 0x555555646390 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xf2390) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #7 0x55555561d3e3 in testing::Test::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xc93e3) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #8 0x55555561dfb4 in testing::TestInfo::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xc9fb4) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #9 0x55555561ea2d in testing::TestSuite::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xcaa2d) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #10 0x55555562fbb1 in testing::internal::UnitTestImpl::RunAllTests() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xdbbb1) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #11 0x55555564fbdb in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xfbbdb) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #12 0x5555556478ec in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xf38ec) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #13 0x55555562dfe4 in testing::UnitTest::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xd9fe4) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #14 0x5555555f6c17 in RUN_ALL_TESTS() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xa2c17) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #15 0x5555555f4cfd in main (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xa0cfd) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #16 0x7ffff3efc1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #17 0x7ffff3efc28a in __libc_start_main_impl ../csu/libc-start.c:360
    #18 0x5555555e9374 in _start (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0x95374) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)

0x51e000008247 is located 3 bytes after 2500-byte region [0x51e000007880,0x51e000008244)
allocated by thread T0 here:
    #0 0x7ffff79cc6c8 in operator new[](unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:98
    #1 0x7ffff6689a81 in nav2_costmap_2d::Costmap2D::initMaps(unsigned int, unsigned int) /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d.cpp:107
    #2 0x7ffff6689c6f in nav2_costmap_2d::Costmap2D::resizeMap(unsigned int, unsigned int, double, double, double) /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d.cpp:118
    #3 0x7ffff669bfb5 in nav2_costmap_2d::LayeredCostmap::resizeMap(unsigned int, unsigned int, double, double, double, bool) /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/layered_costmap.cpp:114
    #4 0x7ffff66b241a in nav2_costmap_2d::Costmap2DROS::on_configure(rclcpp_lifecycle::State const&) /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d_ros.cpp:154
    #5 0x7ffff51faf1e in rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::execute_callback(unsigned int, rclcpp_lifecycle::State const&) const (/opt/ros/rolling/lib/librclcpp_lifecycle.so+0x22f1e) (BuildId: 1169dc85d716b4d26b75c21379b2d8a0065475d1)
    #6 0x7ffff51fb30c in rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::change_state(unsigned char, rclcpp_lifecycle::node_interfaces::LifecycleNodeInterface::CallbackReturn&) (/opt/ros/rolling/lib/librclcpp_lifecycle.so+0x2330c) (BuildId: 1169dc85d716b4d26b75c21379b2d8a0065475d1)
    #7 0x7ffff51fbe8b in rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::trigger_transition(unsigned char) (/opt/ros/rolling/lib/librclcpp_lifecycle.so+0x23e8b) (BuildId: 1169dc85d716b4d26b75c21379b2d8a0065475d1)
    #8 0x5555555f083c in ObstacleFootprint_LineCost_Test::TestBody() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0x9c83c) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #9 0x55555564e8bc in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xfa8bc) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #10 0x555555646390 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xf2390) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #11 0x55555561d3e3 in testing::Test::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xc93e3) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #12 0x55555561dfb4 in testing::TestInfo::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xc9fb4) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #13 0x55555561ea2d in testing::TestSuite::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xcaa2d) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #14 0x55555562fbb1 in testing::internal::UnitTestImpl::RunAllTests() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xdbbb1) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #15 0x55555564fbdb in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xfbbdb) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #16 0x5555556478ec in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xf38ec) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #17 0x55555562dfe4 in testing::UnitTest::Run() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xd9fe4) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #18 0x5555555f6c17 in RUN_ALL_TESTS() (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xa2c17) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #19 0x5555555f4cfd in main (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0xa0cfd) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)
    #20 0x7ffff3efc1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #21 0x7ffff3efc28a in __libc_start_main_impl ../csu/libc-start.c:360
    #22 0x5555555e9374 in _start (/root/nav2_ws/src/navigation2/build/dwb_critics/test/obstacle_footprint_tests+0x95374) (BuildId: 25ff116ec456221f42b844082c3dc40470b40e31)

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d.cpp:266 in nav2_costmap_2d::Costmap2D::getCost(unsigned int, unsigned int) const
Shadow bytes around the buggy address:
  0x51e000007f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x51e000008000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x51e000008080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x51e000008100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x51e000008180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x51e000008200: 00 00 00 00 00 00 00 00[04]fa fa fa fa fa fa fa
  0x51e000008280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51e000008300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51e000008380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51e000008400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x51e000008480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==45344==ABORTING

Expected behavior

Non flaky CI for ObstacleFootprint.LineCost

Actual behavior

ObstacleFootprint.LineCost CI flaky

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions