[BUG] FIXS running slower than real time in co-simulation btw IPG and SUMO

[anyezhou1995]

Describe the bug
Look into what leads to co-simulation running slower than real time as traffic agents scaling up.
The IPG office (with or without Simulink model) will run at the rate of ~0.5x real time as traffic demand gaining up.

To Reproduce
IPG-SUMO:

1. Making sure only one ego vehicle is subscribing information in the 'RS_config_release.yaml' file.
2. Run runI81_zls.bat to get TrafficLayer and SUMO running with proper settings (should be handled in the sumocfg already).
3. Load proper IPG Testrun file, set the proper road, select 'route_62' vehicle route in the road file 'full_stong16'.

IPG-Simulink-SUMO

1. Making sure two ego vehicles are subscribing information in the 'RS_config_release.yaml' file.
2. Open the IPG GUI with 'generic_truck_RS_Control_CompensateRS.mdl' file. May need to put it in CarMaker format...
3. Load proper IPG Testrun scenario as in IPG-SUMO setup.
4. Make sure 'RS files' and 'YAMLMatlab_0.4.3' folders are in the MATLAB PATH.
5. Use the 'sensitivityTest.m' automation script to run the co-simulation.
   <ISSUE_NUMBER>_runningSlowIssue.zip is uploaded to the private OneDrive link:
   Upload to OneDrive (FIXS)

Screenshots (if any)

Environment

• MATLAB 2021b
• SUMO 1.21.0
• IPG TruckMaker 11.0.1

Additional context
Might need some minor tweaks to run in CarMaker as some files are based on TruckMaker.

[yunlishao]

Office version cosimulation speed will become slower and slower. Needs further investigation.

[yunlishao]

Buffer Size Update

Updated send buffer size from 200 to 1024 bytes in commit e0c0de1:

Changes made:

• Increased SENDBUFFERSIZE in RealSimSocket.cpp from 200 to 1024 bytes
• Updated buffer allocations in RealSimPack.m from 200 to 1024 bytes
• Applied changes to all copies in main CommonLib and UserScenarios directories

Rationale:
The previous 200-byte limit was insufficient for messages with all fields enabled. Maximum message size with all fields and max-length strings is approximately 444 bytes:

• Header: 12 bytes
• String fields (7 × 51 bytes max): 357 bytes
• Numeric fields (18 × 4 bytes): 72 bytes
• Single byte fields (3 × 1 byte): 3 bytes

The new 1024-byte buffer provides sufficient headroom for all message configurations.

Files updated:

• CommonLib/RealSimSocket.cpp
• CommonLib/RealSimPack.m
• Corresponding copies in tests/UserScenarios/issue_88_runningSlowIssue/

This should help prevent potential buffer overflow issues that could contribute to performance degradation.

[yunlishao]

Investigation Update

Key Observation: Both IPG office (GUI) and Simulink versions slow down gradually, but when running a standalone CarMaker .exe without RealSim integration, there is no such slowdown issue. Since the standalone version still has GUI, visualization, and APO messaging, this confirms the issue is specifically in the communication/synchronization between User.c and TrafficLayer.exe.

Root Cause Area:
The RealSim integration in User.c, specifically:

• Initialization: VirEnv_initialization() at User_TestRun_Start_atEnd
• RT cycle: VirEnv_runStep(VirEnv_c, SimCore.Time) called every cycle in User_Calc

Likely culprits in VirEnv_runStep:

1. Socket communication buffer accumulation with TrafficLayer
2. Synchronization protocol delays increasing over time
3. Message serialization overhead growing
4. Memory leaks in VirEnvHelper

Next Steps:

1. Add timing instrumentation throughout the custom User_Calc routine

   • Add high-resolution timers before/after VirEnv_runStep()
   • Log execution time every N cycles to track if duration increases with simulation time

2. Test with vanilla CarMaker scenario

   • Use simple CarMaker example scenario (e.g., Brake_Distance, Lane_Change)
   • Same User.c with RealSim integration, but minimal maneuver
   • This will determine if the slowdown is:
     • Scenario-independent → Issue in VirEnv_runStep itself (socket/sync overhead)
     • Scenario-dependent → Issue with complex scenario triggering something in communication layer

3. Inspect VirEnv_Wrapper implementation for socket buffer management, blocking calls, and memory leaks

Expected Outcome: If vanilla CarMaker scenario also slows down, confirms the issue is in the VirEnv_runStep communication layer rather than scenario complexity.

[yunlishao]

Performance Analysis Findings

Based on performance logs from tests/UserScenarios/issue_88_runningSlowIssue/:

Key Metrics (excluding startup)

TrafficLayer (mainTrafficLayer.cpp:917-943):

• traffic_step (SUMO simulation): degrades from ~3.5ms to ~20ms over 126s simulation (1265 steps)
• main_loop: avg 15ms, max 35ms, 75% of loops >10ms

CarMaker (VirEnvHelper.cpp:423-428):

• recv_socket: avg 15ms, max 27ms, 92% of receives >10ms
• runStep_total: avg 0.086ms, max 280ms (startup spike)

Root Cause

Slow SUMO simulation - traffic_step increases ~6x over just ~2 minutes of simulation time, causing CarMaker to block longer on recv_socket waiting for TrafficLayer.

Code Snippets

recv_socket timing (VirEnvHelper.cpp:423-428)

PERF_TIC("recv_socket");
if (Sock_c.recvData(Sock_c.serverSock[iS], &simStateRecv, &simTimeRecv, Msg_c) < 0) {
    *errorMsg = "RealSim: Receive from traffic simulator failed";
    return ERROR_STEP_RECV_REALSIM;
}
PERF_TOC("recv_socket");

traffic_step timing (mainTrafficLayer.cpp:917-943)

PERF_TIC("traffic_step");
if (ENABLE_REALSIM && !g_shutdown.trafficSimulatorClosed) {
    try {
        Traffic_c.runOneStepSimulation();
    }
    catch (const std::exception& e) {
        // error handling...
    }
}
PERF_TOC("traffic_step");

Performance Logs

https://gist.github.com/yunlishao/f4d9dc8e9a0af653242db83ed909caaf
or download here:

• TrafficLayerPerf.log
• RealSimPerf.log

Potential SUMO Optimizations to Investigate

• Multi-threading options
• Asynchronous block-by-block simulation for large networks
• Meso-micro mixed simulation
• Other SUMO performance tuning options