:farmer: `multi_nested_message_instrospection_test` failing consistently in nightly_linux_coverage

[Crola1702]

Bug report

Required Info:

• Operating System:
  • Ubuntu Noble
• Installation type:
  • Source
• Version or commit hash:
  • Rolling and Jazzy

Steps to reproduce issue

1. Run a build in nightly_linux_coverage
2. See projectroot.test_multi_nested_message_introspection test fail

Additional information

Reference build: https://ci.ros2.org/job/nightly_linux_coverage/2426/

Test regressions:

• projectroot.test_multi_nested_message_introspection: rosidl_typesupport_introspection_tests.MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage

Log output:

[ RUN      ] MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage
/home/jenkins-agent/workspace/nightly_linux_coverage/ws/src/ros2/rosidl/rosidl_typesupport_introspection_tests/test/test_multi_nested_message_introspection.cpp:212: Failure
Expected equality of these values:
  item_reference
    Which is: 736-byte object <90-B1 3D-F2 2A-7F 00-00 90-00 00-00 2A-7F 00-00 F0-A7 01-BB 09-56 00-00 F0-A7 01-BB 09-56 00-00 00-00 00-01 0C-00 00-00 35-3C 2E-75 B6-AC 9F-D8 91-85 71-8B 00-00 A1-01 F0-FF FF-FF 00-00 00-01 ... 01-00 00-00 00-00 00-00 60-30 02-BB 09-56 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 80-30 02-BB 09-56 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 A1-01 00-00 00-00>
  message.array_of_arrays[i]
    Which is: 736-byte object <90-B1 3D-F2 2A-7F 00-00 90-B1 3D-F2 2A-7F 00-00 F0-A7 01-BB 09-56 00-00 F0-A7 01-BB 09-56 00-00 00-00 00-01 0C-00 00-00 35-3C 2E-75 B6-AC 9F-D8 91-85 71-8B 00-00 A1-01 F0-FF FF-FF 00-00 00-01 ... 01-00 00-00 00-00 00-00 00-C3 01-BB 09-56 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 E0-C2 01-BB 09-56 00-00 09-00 00-00 00-00 00-00 0A-00 00-00 00-00 00-00 00-00 A1-01 10-00 00-00>

/home/jenkins-agent/workspace/nightly_linux_coverage/ws/src/ros2/rosidl/rosidl_typesupport_introspection_tests/test/test_multi_nested_message_introspection.cpp:256: Failure
Expected equality of these values:
  message
    Which is: 6864-byte object <90-B1 3D-F2 2A-7F 00-00 90-B1 3D-F2 2A-7F 00-00 F0-A7 01-BB 09-56 00-00 F0-A7 01-BB 09-56 00-00 00-00 00-01 0C-00 00-00 35-3C 2E-75 B6-AC 9F-D8 91-85 71-8B 00-00 A1-01 F0-FF FF-FF 00-00 00-01 ... 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 20-FA 01-BB 09-56 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 70-FF 01-BB 09-56 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00>
  message_copy
    Which is: 6864-byte object <90-B1 3D-F2 2A-7F 00-00 90-00 00-00 2A-7F 00-00 F0-A7 01-BB 09-56 00-00 F0-A7 01-BB 09-56 00-00 00-00 00-01 0C-00 00-00 35-3C 2E-75 B6-AC 9F-D8 91-85 71-8B 00-00 A1-01 F0-FF FF-FF 00-00 00-01 ... 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 70-85 02-BB 09-56 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00 10-8D 02-BB 09-56 00-00 01-00 00-00 00-00 00-00 01-00 00-00 00-00 00-00>

[  FAILED  ] MultiNestedMessageIntrospectionTest/0.CanWriteTypeErasedMessage, where TypeParam = rosidl_typesupport_introspection_tests__msg__MultiNested (1 ms)

Flakiness ratio:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_coverage	2024-06-14	2024-05-30	15	15	100.0
nightly_win_rep	2024-06-13	2024-06-05	12	2	16.67

[fujitatomoya]

/assign @ahcorde

[ahcorde]

I compiled ros2 rolling with the mixin coverage-gcc but I'm not able to reproduce this error locally

• coverage-gcc mixin

@Crola1702 @mjcarroll any ideas about how to reproduce this locally ?

[Blast545]

I don't think this is the most straightforward way to reproduce it, but I just installed the mixins and ran the same build and test commands that are run in CI, and was able to reproduce the error.

Those are the commands that I used, in case it helps:

colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml
colcon mixin update default

colcon build --base-paths "src" --build-base "build" --install-base "install" --event-handlers console_cohesion+ console_package_list+ --cmake-args -DBUILD_TESTING=ON --no-warn-unused-cli -DCMAKE_BUILD_TYPE=Debug -DINSTALL_EXAMPLES=OFF -DSECURITY=ON -DAPPEND_PROJECT_NAME_TO_INCLUDEDIR=ON --packages-up-to action_msgs ament_index_cpp builtin_interfaces class_loader composition_interfaces console_bridge_vendor diagnostic_msgs fastcdr fastrtps foonathan_memory_vendor geometry_msgs libstatistics_collector libyaml_vendor lifecycle_msgs nav_msgs rcl rcl_action rcl_interfaces rcl_lifecycle rcl_logging_spdlog rcl_yaml_param_parser rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle rcpputils rcutils rmw rmw_dds_common rmw_fastrtps_cpp rmw_fastrtps_shared_cpp rmw_implementation rosgraph_msgs rosidl_default_runtime rosidl_runtime_c rosidl_runtime_cpp rosidl_typesupport_c rosidl_typesupport_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_typesupport_interface rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp spdlog_vendor statistics_msgs std_msgs std_srvs tracetools trajectory_msgs unique_identifier_msgs visualization_msgs interactive_markers launch_testing_ros message_filters ros2action ros2component ros2doctor ros2interface ros2lifecycle ros2lifecycle_test_fixtures ros2param ros2topic rosbag2_compression rosbag2_cpp rosbag2_storage rosbag2_storage_default_plugins rosbag2_test_common rosbag2_tests rosbag2_transport rosidl_generator_c rosidl_generator_cpp rosidl_generator_py rosidl_runtime_py rosidl_typesupport_introspection_tests test_cli test_cli_remapping test_communication test_launch_ros test_msgs test_quality_of_service test_rclcpp test_security test_tf2 test_tracetools tf2 tf2_bullet tf2_eigen tf2_geometry_msgs tf2_kdl tf2_msgs tf2_py tf2_ros tf2_sensor_msgs --ament-cmake-args -DCMAKE_CXX_FLAGS="${CMAKE_CXX_FLAGS} --coverage" -DCMAKE_C_FLAGS="${CMAKE_C_FLAGS} --coverage"

colcon test --base-paths "src" --build-base "build" --install-base "install" --pytest-with-coverage --event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-up-to action_msgs ament_index_cpp builtin_interfaces class_loader composition_interfaces console_bridge_vendor diagnostic_msgs fastcdr fastrtps foonathan_memory_vendor geometry_msgs libstatistics_collector libyaml_vendor lifecycle_msgs nav_msgs rcl rcl_action rcl_interfaces rcl_lifecycle rcl_logging_spdlog rcl_yaml_param_parser rclcpp rclcpp_action rclcpp_components rclcpp_lifecycle rcpputils rcutils rmw rmw_dds_common rmw_fastrtps_cpp rmw_fastrtps_shared_cpp rmw_implementation rosgraph_msgs rosidl_default_runtime rosidl_runtime_c rosidl_runtime_cpp rosidl_typesupport_c rosidl_typesupport_cpp rosidl_typesupport_fastrtps_c rosidl_typesupport_fastrtps_cpp rosidl_typesupport_interface rosidl_typesupport_introspection_c rosidl_typesupport_introspection_cpp spdlog_vendor statistics_msgs std_msgs std_srvs tracetools trajectory_msgs unique_identifier_msgs visualization_msgs interactive_markers launch_testing_ros message_filters ros2action ros2component ros2doctor ros2interface ros2lifecycle ros2lifecycle_test_fixtures ros2param ros2topic rosbag2_compression rosbag2_cpp rosbag2_storage rosbag2_storage_default_plugins rosbag2_test_common rosbag2_tests rosbag2_transport rosidl_generator_c rosidl_generator_cpp rosidl_generator_py rosidl_runtime_py rosidl_typesupport_introspection_tests test_cli test_cli_remapping test_communication test_launch_ros test_msgs test_quality_of_service test_rclcpp test_security test_tf2 test_tracetools tf2 tf2_bullet tf2_eigen tf2_geometry_msgs tf2_kdl tf2_msgs tf2_py tf2_ros

[Blast545]

The errors is happening in rosidl_typesupport_introspection_tests, so I imagine this should suffice:

colcon mixin add default https://raw.githubusercontent.com/colcon/colcon-mixin-repository/master/index.yaml
colcon mixin update default

colcon build --base-paths "src" --build-base "build" --install-base "install" --event-handlers console_cohesion+ console_package_list+ --cmake-args -DBUILD_TESTING=ON --no-warn-unused-cli -DCMAKE_BUILD_TYPE=Debug -DINSTALL_EXAMPLES=OFF -DSECURITY=ON -DAPPEND_PROJECT_NAME_TO_INCLUDEDIR=ON --packages-up-to rosidl_typesupport_introspection_tests --ament-cmake-args -DCMAKE_CXX_FLAGS="${CMAKE_CXX_FLAGS} --coverage" -DCMAKE_C_FLAGS="${CMAKE_C_FLAGS} --coverage"

colcon test --base-paths "src" --build-base "build" --install-base "install" --pytest-with-coverage --event-handlers console_cohesion+ --retest-until-pass 2 --ctest-args -LE xfail --pytest-args -m "not xfail" --packages-up-to rosidl_typesupport_introspection_tests

[Blast545]

🧑‍🌾 @ahcorde friendly ping

[Crola1702]

Friendly ping @ahcorde