Low Frame Rate Issue with Concurrent Lidar and Camera

[HandsomeAIccx]

Version: Apollo 9.0 Hardware: X86 and Orin
Issue Details: I am experiencing an issue where launching lidar and camera perception using the same launch file results in the lidar frame rate dropping from the original 10 frames to only 2 or 3 frames. However, if I use two separate launch files to start the lidar and camera simultaneously, the frame rate remains normal.

Does this indicate that running camera perception and lidar perception in the same process leads to GPU resource contention, whereas separate processes do not encounter this issue?

Currently, I want to perform multi-sensor fusion, such as with perception_all.launch, which starts lidar perception and camera perception within the same process. I wish to separate these two. But because the channels sent by the them are internal channels, they cannot be found in cyber_monitor. Therefore, to start multi_sensor_fusion, it seems I have to run it in the same launch file, but the frame rate is poor. What should I do? Is it possible to use msg_adapter to do something helpful? Are there any examples available?

[daohu527]

What should I do?

Yes, the lidar, camera and fusion modules need to be started at the same process(one launch), the reason is just as you said "because the channels they send are internal channels"

Is it possible to use msg_adapter to do something helpful?

Yes, you can use some msg_adapter to publish messages, note that messages have serialization methods, and "msg_adapter" needs the same processing as lidar, camera and fusion modules

issue

lidar frame rate dropping from the original 10 frames to only 2 or 3 frames

I'm not sure what the problem is, simply you can add logs and check if the lidar module frequency is 10 hz, then we can narrow down the problem.

By the way, you can use the following method to collect the module time cost

PERF_BLOCK("filter_bank") 

It will save all the logs in a file called "PERF.info.log", then you can use aperf tool to visualize the time cost line.

https://github.com/ApolloAuto/apollo/blob/master/modules/tools/aperf/README.md

[HandsomeAIccx]

@daohu527
Thanks. I failed to find the PERF.info.log. But I printed the time of entering and exiting for each component in
time_cost.txt
It is clear that the lidar_detection component is slower than any other front component, while the other front components seem to be fine. (It always takes more than 200 ms.)

The issue only occurs when I launch camera_detection_multi_stage_yolox3d.dag and lidar_detection.dag in the same launch file.

If I use two separate launch files to launch them individually at the same time, the time for lidar_detection is about 100 ms.

But if I want to use fusion, the issue seems to be unavoidable because I need to launch them in the same file.
Can you give me any suggestions? Thank you

What should I do?

Yes, the lidar, camera and fusion modules need to be started at the same process(one launch), the reason is just as you said "because the channels they send are internal channels"

Is it possible to use msg_adapter to do something helpful?

Yes, you can use some msg_adapter to publish messages, note that messages have serialization methods, and "msg_adapter" needs the same processing as lidar, camera and fusion modules

issue

lidar frame rate dropping from the original 10 frames to only 2 or 3 frames

I'm not sure what the problem is, simply you can add logs and check if the lidar module frequency is 10 hz, then we can narrow down the problem.

By the way, you can use the following method to collect the module time cost

PERF_BLOCK("filter_bank") 

It will save all the logs in a file called "PERF.info.log", then you can use aperf tool to visualize the time cost line.

https://github.com/ApolloAuto/apollo/blob/master/modules/tools/aperf/README.md

[daohu527]

which Apollo version do you use? I check the code default perf.log is on!

apollo/scripts/apollo_base.sh

Line 43 in ba32187

ENABLE_PROFILER=true

The reason why this is used is that it can record the time of each module, So I suggest you use aperf to analyze!

[HandsomeAIccx]

@daohu527
I m in Tags :v9.0.0,looks like the default perf.log is off.

[daohu527]

@HandsomeAIccx You can refer to modify and add ENABLE_PROFILER, or format your time consumption, your log module is not aligned