MPPI crashing on loading plug-ings

[AmmarAlbakri]

Bug report

• Operating System:
  • Ubuntu 22.04
• ROS2 Version:
• Humble Source

After the binary packages sync on 18.08.2023 the nav2_mppi_controller started crashing on when launching;

ps: same non-updated setup still working.

By the way, the sync was supposed to fix this issue but it hasn't been solved yet: #3762 (comment)

[SteveMacenski]

This is another issue. Please compile with debug flags according to the docs here (https://navigation.ros.org/tutorials/docs/get_backtrace.html) and give me the traceback to know where the crash is occurring. I looked over the recent changes and I can't find anything obvious that would cause this. Make sure that you clean build (e.g. delete install/build spaces) in case there's some caching issue causing this from a major sync update

[AmmarAlbakri]

I was able to get this backtrace when running mppi from the binary packages:

[INFO] [1692688226.327340653] [controller_server]: Created controller : FollowPath of type nav2_rotation_shim_controller::RotationShimController
[INFO] [1692688226.349984032] [controller_server]: Created internal controller for rotation shimming: FollowPath of type nav2_mppi_controller::MPPIController
[WARN] [1692688226.351220055] [controller_server]: Controller period is less then model dt, consider setting it equal
[INFO] [1692688226.357645621] [controller_server]: ConstraintCritic instantiated with 1 power and 4.000000 weight.
[INFO] [1692688226.357812764] [controller_server]: Critic loaded : mppi::critics::ConstraintCritic

Thread 1 "controller_serv" received signal SIGILL, Illegal instruction.
0x00007fffcb5b9e15 in mppi::critics::ObstaclesCritic::findCircumscribedCost(std::shared_ptr<nav2_costmap_2d::Costmap2DROS>)

I hope this helps.

When I build from source on humble branch there is no problem everything works like a charm.

[SteveMacenski]

What is your computer architecture? That is also not a traceback, that doesn't show me where the issue is in particular. I still don't see anything obviously wrong to address.

If you compile it from source from the 1.1.9 tag (humble which is the exact thing that should be in binaries) does that work for you?

[AmmarAlbakri]

My computer architecutre is:

6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

I am sorry I don't have experience with gdp or backtrace, I followed the tutorial

I compiled from source 1.1.9 tag, and it is working, but the binaries still give the error I mentioned above. I also tried it on many different computer architectures but still, binaries throw an error and crash.

[StetroF]

I met this problem as well. I found that this controller can't work in the edition that downloaded by 'sudo apt install ros-humble-nav2-mppi-controller'. And solve it by build the nav2_mppi_controller’s source code rather than download it by sudo. Perhaps it is because the sudo edition didn't update?

[SteveMacenski]

This is really odd. What happens if you sudo apt purge ros-humble-nav2-mppi-controller and reinstall it? I'm wondering if something is broken in the update process more than the binaries being broken.

I looked into this in detail and I don't see any changes in the function that @AmmarAlbakri got a trace pointing to. Nothing changed with respect to that function or its inputs -- but things around it did change. The rest of Humble and the sync look identical to me, so if its working on local builds but not in binary format I can only seeing it be the cause of 3 things

• Binaries are busted
• Binary install is busted
• The build on the build farm makes binaries that don't run on related but non-identical computers for some reason (but worked before was a fluke?)

The second item is the one I can have you guys easily test. The first I've asked Intrinsic if we can re-trigger to test. The last is more complex and I don't want to go down that rabbit hole if unnecessary.

Please let me know!

[StetroF]

Once i purge nav2-mppi-controller.The whole navigation will be remove so i reinstall them by sudo apt install ros-humble-navigation2. Then I relaunch my nav2 package. The same problem occur. After i add the builded nav2-mppi-controller's code in my install package. It work again.
Last, Here is the relative log when loading the mppi controller installed by sudo rather than source code build:

[controller_server-15] [INFO] [1693269845.941573472] [controller_server]: Created controller : FollowPath of type nav2_mppi_controller::MPPIController
[controller_server-15] [WARN] [1693269845.950492480] [controller_server]: Controller period is less then model dt, consider setting it equal
[controller_server-15] [INFO] [1693269845.953610926] [controller_server]: ConstraintCritic instantiated with 1 power and 4.000000 weight.
[controller_server-15] [INFO] [1693269845.953682884] [controller_server]: Critic loaded : mppi::critics::ConstraintCritic
[controller_server-15] [INFO] [1693269845.955963772] [controller_server]: GoalCritic instantiated with 1 power and 5.000000 weight.
[controller_server-15] [INFO] [1693269845.956013774] [controller_server]: Critic loaded : mppi::critics::GoalCritic
[controller_server-15] [INFO] [1693269845.957338102] [controller_server]: GoalAngleCritic instantiated with 1 power, 3.000000 weight, and 0.500000 angular threshold.
[controller_server-15] [INFO] [1693269845.957417906] [controller_server]: Critic loaded : mppi::critics::GoalAngleCritic
[controller_server-15] [INFO] [1693269845.960066387] [controller_server]: ReferenceTrajectoryCritic instantiated with 1 power and 14.000000 weight
[controller_server-15] [INFO] [1693269845.960157092] [controller_server]: Critic loaded : mppi::critics::PathAlignCritic
[controller_server-15] [INFO] [1693269845.962222725] [controller_server]: Critic loaded : mppi::critics::PathFollowCritic
[robot_pose_publisher-8] Warning: Invalid frame ID "map" passed to canTransform argument target_frame - frame does not exist
[robot_pose_publisher-8] at line 93 in ./src/buffer_core.cpp
[controller_server-15] [INFO] [1693269845.966025416] [controller_server]: PathAngleCritic instantiated with 1 power and 2.000000 weight. Reversing allowed.
[controller_server-15] [INFO] [1693269845.966077558] [controller_server]: Critic loaded : mppi::critics::PathAngleCritic
[controller_server-15] [INFO] [1693269845.966705911] [controller_server]: PreferForwardCritic instantiated with 1 power and 2.000000 weight.
[controller_server-15] [INFO] [1693269845.966739201] [controller_server]: Critic loaded : mppi::critics::PreferForwardCritic

Then after the preferforwardcritic loaded .The error occur:
[ERROR] [controller_server-15]: process has died [pid 13909, exit code -4, cmd '/opt/ros/humble/lib/nav2_controller/controller_server --ros-args --log-level info --ros-args -r _ --params-file /tmp/tmpsaf6nhx7 -r /tf:=tf -r /tf_static:=tf_static -r cmd_vel:=cmd_vel_nav'].

As you see . Controller server crashed after loading the critic.

[SteveMacenski]

Can you re-order / remove some of the critics? Is there one in particular that makes it crash? Someone else got a backtrace that pointed to the Obstacle critic -- but maybe its not that simple. Try with 0 critics and work up from there one by one.

If purging and reinstalling doesn't work, I don't think its the debian update process that's the issue. The binary bin job looks fine to me in build.ros2.org so I'm not sure what could cause the issue there. @StetroF what's your computer architecture?

[StetroF]

I tried. First time , I delete the constraincritic and obstaclecritic, then i find that it stil crashed. But i found it there are some logs relative with PathAngleCritic's intialize. So I thought this critic must be normal.Then second time i delete all critic except PathAngleCritic. But this time after loading the critic, the logs were:

[controller_server-15] [INFO] [1693354275.581728814] [sto_robot2.controller_server]: Created controller : FollowPath of type nav2_mppi_controller::MPPIController
[controller_server-15] [WARN] [1693354275.583152778] [sto_robot2.controller_server]: Controller period is less then model dt, consider setting it equal
[controller_server-15] [INFO] [1693354275.585134635] [sto_robot2.controller_server]: PathAngleCritic instantiated with 1 power and 2.000000 weight. Reversing allowed.
[controller_server-15] [INFO] [1693354275.585179252] [sto_robot2.controller_server]: Critic loaded : mppi::critics::PathAngleCritic
[controller_server-15] [INFO] [1693354275.587878907] [sto_robot2.controller_server]: Optimizer reset

Then it died again.I've tried other options that launch with only 1 critic like GoalAngleCritic ,GoalCritic .etc. The logs are similar with the logs upon that after the 'critic loaded ...... ' and 'Optimizer reset' then it crashed.
Last, I delete all critic, the log were :

[controller_server-15] [INFO] [1693354834.866720886] [sto_robot2.controller_server]: Created controller : FollowPath of type nav2_mppi_controller::MPPIController
[controller_server-15] [WARN] [1693354834.873562877] [sto_robot2.controller_server]: Controller period is less then model dt, consider setting it equal
[controller_server-15] [INFO] [1693354834.878223025] [sto_robot2.controller_server]: Optimizer reset

Then crashed again.Last, my compute architecture is :

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian

[SteveMacenski]

Fun.

[sto_robot2.controller_server]: Optimizer reset

That's the last line of the optimizer's initialization in the bringup https://github.com/ros-planning/navigation2/blob/210977790bfb6e809f2c2ef3b7e1660662d630cc/nav2_mppi_controller/src/optimizer.cpp#L54

The very next thing is the path handler initialization. The following lines were add - but all are in the 1.1.9 sync that was backported to humble

  getParam(enforce_path_inversion_, "enforce_path_inversion", false);
  if (enforce_path_inversion_) {
    getParam(inversion_xy_tolerance_, "inversion_xy_tolerance", 0.2);
    getParam(inversion_yaw_tolerance, "inversion_yaw_tolerance", 0.4);
    inversion_locale_ = 0u;
  }

I'm morbidly curious if you add those parameters to your parameter file for MPPI if that crash goes away (e.g. something going on with missing parameters). Keep all the critics out of the configuration. I doubt it'll do anything, but if it does that would be extremely relevant to know.

Also, getting a backtrace for the crash that occurs when you're not using any critics would be another valuable data point https://navigation.ros.org/tutorials/docs/get_backtrace.html. Especially if you're not seeing the exact same behavior / issue as @AmmarAlbakri

What seems odd to me is that both of these areas of code were changed recently. But neither have been changed since 1.1.9 was released that you've mentioned you've cloned that tag which is the exact same software and seems to work. It makes me think something's up in buildfarm-land

Edit:

Another thing to test is https://github.com/ros-planning/navigation2/blob/main/nav2_mppi_controller/CMakeLists.txt#L38-L47

It seems that the illegal instruction complaint might be that those flags are illegal for your CPUs but legal for the build farm. If you change

add_compile_options(-O3 -finline-limit=1000000 -ffp-contract=fast -ffast-math)

to

add_compile_options(-O3 -finline-limit=1000000 -ffp-contract=fast -ffast-math -mavx2 -mfma)

And compile locally, do you see any change in behavior with the locally compiled version (e.g. compile fail, or the same binary related issue)? I want both @AmmarAlbakri and @StetroF to try this since you may not actually be experiencing the same issue

[MichalMalachowski]

@SteveMacenski I have the same problem, I use ROS Humble Nav2 stack inside Docker container. MPPI controller is proven to launch correctly in the container created the second-to-last Friday (18th August) and it doesn't launch in the container created yesterday (29th August).

I tried some of advises you gave in the last reply:

• empty critics list
• added parameters enforce_path_inversion: false, inversion_xy_tolerance: 0.2, inversion_yaw_tolerance: 0.4 for the controller in config yaml
• tested with both my configuration and an example configuration from Nav2 website

None combination of above worked.

I didn't try to compile the package locally. I could play with it when I find some free time, but let's hope someone else will check it before.

[SteveMacenski]

None combination of above worked.

Any traceback to what method specifically crashed? This would be helpful, especially if it changed between different runs / things.

I didn't try to compile the package locally.

Please do. I'm playing with this right now but I only have about 2 hours today to spend on it. Especially if the types of crashes are different for different people's computers, I'll need that data to know if I'm barking up the right tree from another source. I see this now that I'm testing it locally, which is good that I can reproduce and experiment myself. Though difficult since it only appears in binaries :(

[SteveMacenski]

Notes:

• Compiled locally OK
• Compiled locally with forcing on -mavx2 -mfma OK
• Compiled locally with forcing off -mavx2 -mfma OK

---

• Binaries with the Configuration Docs configs NOT OK

> const&, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>, mppi::ParametersHandler*) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#4  0x00007ffff40fbde4 in mppi::Optimizer::initialize(std::weak_ptr<rclcpp_lifecycle::LifecycleNode>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>, mppi::ParametersHandler*) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#5  0x00007ffff40da0c1 in nav2_mppi_controller::MPPIController::configure(std::weak_ptr<rclcpp_lifecycle::LifecycleNode> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<tf2_ros::Buffer>, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#6  0x00007ffff7cade9d in nav2_controller::ControllerServer::on_configure(rclcpp_lifecycle::State const&) ()
   from /home/steve/Documents/hi_ws/install/nav2_controller/lib/libcontroller_server_core.so
#7  0x00007ffff7dacc6b in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#8  0x00007ffff7db66d2 in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#9  0x00007ffff7da4730 in rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::on_change_state(std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >) () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#10 0x00007ffff7da6936 in std::_Function_handler<void (std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >), std::_Bind<void (rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::*(rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<rmw_request_id_s>&&, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >&&, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >&&) () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#11 0x00007ffff7db4819 in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#12 0x00007ffff7eae046 in ?? () from /opt/ros/humble/lib/librclcpp.so
#13 0x00007ffff7eaba2a in rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>) () from /opt/ros/humble/lib/librclcpp.so
#14 0x00007ffff7eabd96 in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) () from /opt/ros/humble/lib/librclcpp.so
#15 0x00007ffff7eb3620 in rclcpp::executors::SingleThreadedExecutor::spin() () from /opt/ros/humble/lib/librclcpp.so
#16 0x00007ffff7eb3835 in rclcpp::spin(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>) () from /opt/ros/humble/lib/librclcpp.so

---

• Binaries with Arbitrary critics NOT OK

w/ Obstacle critic its always

(gdb) backtrace
#0  0x00007ffff4061d56 in mppi::critics::ObstaclesCritic::findCircumscribedCost(std::shared_ptr<nav2_costmap_2d::Costmap2DROS>) ()
   from /opt/ros/humble/lib/libmppi_critics.so
#1  0x00007ffff4062501 in mppi::critics::ObstaclesCritic::initialize() ()
   from /opt/ros/humble/lib/libmppi_critics.so
#2  0x00007ffff4122dc9 in mppi::CriticManager::loadCritics() ()
   from /opt/ros/humble/lib/libmppi_controller.so
#3  0x00007ffff411d2b1 in mppi::CriticManager::on_configure(std::weak_ptr<rclcpp_lifecycle::LifecycleNode>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>, mppi::ParametersHandler*) () from /opt/ros/humble/lib/libmppi_controller.so
#4  0x00007ffff40fbde4 in mppi::Optimizer::initialize(std::weak_ptr<rclcpp_lifecycle::LifecycleNode>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>, mppi::ParametersHandler*) () from /opt/ros/humble/lib/libmppi_controller.so
#5  0x00007ffff40da0c1 in nav2_mppi_controller::MPPIController::configure(std::weak_ptr<rclcpp_lifecycle::LifecycleNode> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<tf2_ros::Buffer>, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#6  0x00007ffff7cade9d in nav2_controller::ControllerServer::on_configure(rclcpp_lifecycle::State const&) ()
   from /home/steve/Documents/hi_ws/install/nav2_controller/lib/libcontroller_se

W/ goal angle, or Prefer forward critic only, Constraint critic only, or path follow only, or path angle only, or path align only, or goal angle only, or twirling only: it switches between one of these two on different runs (which seems reasonable b/c the NoiseGenerator is in another thread, so its basically "who triggers the failure first)

/opt/ros/humble/lib/libmppi_controller.so
(gdb) backtrace
#0  0x00007ffff412fc82 in mppi::NoiseGenerator::generateNoisedControls() () from /opt/ros/humble/lib/libmppi_controller.so
#1  0x00007ffff41308d3 in mppi::NoiseGenerator::noiseThread() () from /opt/ros/humble/lib/libmppi_controller.so
#2  0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff7494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007ffff7526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

#0  0x00007ffff4129e3b in mppi::PathHandler::getMaxCostmapDist() ()
   from /opt/ros/humble/lib/libmppi_controller.so
#1  0x00007ffff4129f4b in mppi::PathHandler::initialize(std::weak_ptr<rclcpp_lifecycle::LifecycleNode>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>, std::shared_ptr<tf2_ros::Buffer>, mppi::ParametersHandler*) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#2  0x00007ffff40da157 in nav2_mppi_controller::MPPIController::configure(std::weak_ptr<rclcpp_lifecycle::LifecycleNode> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<tf2_ros::Buffer>, std::shared_ptr<nav2_costmap_2d::Costmap2DROS>) ()
   from /opt/ros/humble/lib/libmppi_controller.so
#3  0x00007ffff7cade9d in nav2_controller::ControllerServer::on_configure(rclcpp_lifecycle::State const&) ()
   from /home/steve/Documents/hi_ws/install/nav2_controller/lib/libcontroller_server_core.so
#4  0x00007ffff7dacc6b in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#5  0x00007ffff7db66d2 in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#6  0x00007ffff7da4730 in rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::on_change_state(std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >) ()
   from /opt/ros/humble/lib/librclcpp_lifecycle.so
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x00007ffff7da6936 in std::_Function_handler<void (std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >), std::_Bind<void (rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl::*(rclcpp_lifecycle::LifecycleNode::LifecycleNodeInterfaceImpl*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(std::shared_ptr<rmw_request_id_s>, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<rmw_request_id_s>&&, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Request_<std::allocator<void> > >&&, std::shared_ptr<lifecycle_msgs::srv::ChangeState_Response_<std::allocator<void> > >&&) () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#8  0x00007ffff7db4819 in ?? () from /opt/ros/humble/lib/librclcpp_lifecycle.so
#9  0x00007ffff7eae046 in ?? () from /opt/ros/humble/lib/librclcpp.so
#10 0x00007ffff7eaba2a in rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>) ()
   from /opt/ros/humble/lib/librclcpp.so
#11 0x00007ffff7eabd96 in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) ()
   from /opt/ros/humble/lib/librclcpp.so
#12 0x00007ffff7eb3620 in rclcpp::executors::SingleThreadedExecutor::spin() ()
   from /opt/ros/humble/lib/librclcpp.so
#13 0x00007ffff7eb3835 in rclcpp::spin(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>) ()
   from /opt/ros/humble/lib/librclcpp.so
#14 0x00005555555563c0 in main ()

Both of which complain of an illegal instruction where the obstacle layer's seg faults outright.

That seems to point to the ordering:

• Obstacle, when present, will trigger first
• If not, then either the noise generator or the path handler, depending on which thread can get there first

---

So lets look at these functions

double PathHandler::getMaxCostmapDist()
{
  const auto & costmap = costmap_->getCostmap();
  return std::max(costmap->getSizeInCellsX(), costmap->getSizeInCellsY()) *
         costmap->getResolution() / 2.0;
}

void NoiseGenerator::generateNoisedControls()
{
  auto & s = settings_;

  xt::noalias(noises_vx_) = xt::random::randn<float>(
    {s.batch_size, s.time_steps}, 0.0,
    s.sampling_std.vx);
  xt::noalias(noises_wz_) = xt::random::randn<float>(
    {s.batch_size, s.time_steps}, 0.0,
    s.sampling_std.wz);
  if (is_holonomic_) {
    xt::noalias(noises_vy_) = xt::random::randn<float>(
      {s.batch_size, s.time_steps}, 0.0,
      s.sampling_std.vy);
  }
}

double ObstaclesCritic::findCircumscribedCost(
  std::shared_ptr<nav2_costmap_2d::Costmap2DROS> costmap)
{
  double result = -1.0;
  bool inflation_layer_found = false;
  // check if the costmap has an inflation layer
  for (auto layer = costmap->getLayeredCostmap()->getPlugins()->begin();
    layer != costmap->getLayeredCostmap()->getPlugins()->end();
    ++layer)
  {
    auto inflation_layer = std::dynamic_pointer_cast<nav2_costmap_2d::InflationLayer>(*layer);
    if (!inflation_layer) {
      continue;
    }

    inflation_layer_found = true;
    const double circum_radius = costmap->getLayeredCostmap()->getCircumscribedRadius();
    const double resolution = costmap->getCostmap()->getResolution();
    result = inflation_layer->computeCost(circum_radius / resolution);
    auto getParam = parameters_handler_->getParamGetter(name_);
    getParam(inflation_scale_factor_, "cost_scaling_factor", 10.0);
    getParam(inflation_radius_, "inflation_radius", 0.55);
  }

  if (!inflation_layer_found) {
    RCLCPP_WARN(
      rclcpp::get_logger("computeCircumscribedCost"),
      "No inflation layer found in costmap configuration. "
      "If this is an SE2-collision checking plugin, it cannot use costmap potential "
      "field to speed up collision checking by only checking the full footprint "
      "when robot is within possibly-inscribed radius of an obstacle. This may "
      "significantly slow down planning times and not avoid anything but absolute collisions!");
  }

  return result;
}

---

What didn't change anything

• Messing with the controller speed
• Messing with key params for data sizing (in case running out of memory) like batch size, time steps etc
• removing all params and removing all critic params

--> So I'm pretty sure it has nothing to do with configurations being broken.

My local xtensor version is 0.23.10-15 which is years old, so that wasn't recently updated. I also checked the build farm and its using the same version. I also scanned over all the logs and nothing looks out of the ordinary.

I just tested and the iron binaries are broken too. Beyond the fact that these are orbiting pieces of code that have changed recently, I can't find any throughline or why this would be happening only in the binaries but not in source code -- nor why anything has illegal instructions.

[AmmarAlbakri]

And compile locally, do you see any change in behavior with the locally compiled version (e.g. compile fail, or the same binary related issue)? I want both @AmmarAlbakri and @StetroF to try this since you may not actually be experiencing the same issue

I have tested compiling with -mavx2 -mfma flag and it works as long as I am building locally. Binaries still give the same error.