'Error: Timeout: Agent took too long to setup' and other issues

[hae-sung-oh]

Hi author,
I greatly appreciate your fantastic work!
However, while attempting to implement the repository on my local machine, I’ve encountered a few issues.
I would appreciate it if you could provide some guidance.
Thank you!

Environments:
i7-8700K
3080Ti + 12GM VRAM
64GB RAM
Ubuntu 20.04
CARLA 0.9.12 (ported successfully from 0.9.10)
CARLA scenario_runner 0.9.12
python 3.8

---

1. Game time / System time ratio is remarkably low: 0.2
In other words, the simulation is running too slowly.
Despite having reasonably adequate hardware, performance remains awful.
I monitored resource usage during the simulation, and the highest GPU and CPU usages are only 30% and 20%, respectively.

Notably, only a single CPU core is being utilized at any given time, which seems to be causing a bottleneck.
Do you have any suggestions for enabling parallel CPU computation to improve performance?

---

2. The performance of the provided pre-trained model is not high enough.
With the given model pth file, the average RouteCompletionTest score is up to 40% on my PC.
As I mentioned before, the hardware spec seems not to be a reason for it.
Is there any possible way to improve it?

---

3. Phantom obstacles
The pre-trained model detects un-existing obstacles and stops to avoid it.
Observed situations are the below:

• False red light (red_light_prob = 1)
• False pedestrian (The model refers vending machines as pedestrians, and stops)
• Empty objects
  Is there any idea to improve the model without training from the scratch?

---

4. Error: Agent took too long to set up
After 3 or 4 data generation iterations, the following error occurs and the simulation crashes:

Error during the simulation:
> Timeout: Agent took too long to setup

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 353, in _load_and_run_scenario
    self.manager.run_scenario()
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in run_scenario
    self._tick_scenario(timestamp)
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/scenarios/scenario_manager.py", line 181, in _tick_scenario
    CarlaDataProvider.get_world().tick(self._timeout)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 123, in _signal_handler
    raise RuntimeError("Timeout: Agent took too long to setup")
RuntimeError: Timeout: Agent took too long to setup
> Stopping the route

Watchdog exception - Timeout of 1001.0 seconds occured

The scenario could not be loaded:
> Timeout: Agent took too long to setup

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 314, in _load_and_run_scenario
    self._load_and_wait_for_world(args, config.town, config.ego_vehicles)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 213, in _load_and_wait_for_world
    self.world = self.client.load_world(town)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 123, in _signal_handler
    raise RuntimeError("Timeout: Agent took too long to setup")
RuntimeError: Timeout: Agent took too long to setup

Watchdog exception - Timeout of 1001.0 seconds occured
Traceback (most recent call last):
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/scenarios/scenario_manager.py", line 152, in _tick_scenario
    ego_action = self._agent()
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 82, in __call__
    return self._agent()
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 104, in __call__
    input_data = self.sensor_interface.get_data()
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/envs/sensor_interface.py", line 244, in get_data
    sensor_data = self._new_data_buffers.get(True, self._queue_timeout)
  File "/usr/lib/python3.8/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/usr/lib/python3.8/threading.py", line 306, in wait
    gotit = waiter.acquire(True, timeout)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 123, in _signal_handler
    raise RuntimeError("Timeout: Agent took too long to setup")
RuntimeError: Timeout: Agent took too long to setup

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 469, in main
    leaderboard_evaluator.run(arguments)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 417, in run
    self._load_and_run_scenario(args, config)
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 353, in _load_and_run_scenario
    self.manager.run_scenario()
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/scenarios/scenario_manager.py", line 136, in run_scenario
    self._tick_scenario(timestamp)
  File "/home/ohs-dyros/gitRepo/InterFuser/leaderboard/leaderboard/scenarios/scenario_manager.py", line 159, in _tick_scenario
    raise AgentError(e)

I have tried:

• Increasing timeout value
• Adding 'try except' lines in the 'while' loop
• Running on a single server only
• Re-initializing from the scratch on every iterations
  but none of the above could not resolve the error.

What is the cause of these errors?
Is the version of CARLA matters?
Any advises for resolving must be very helpful.

---

Thank you for your answers in advance!
Sincerely, Haesung.

[deepcs233]

Hi!
Q1: Maybe you need to refer to Carla's Github Repo. The performance of the Carla simulator is not optimized well.
Q2: Can you provide more details? For example, which model, which route, which scenario?
Q3: This issue can help you: #71 I think more correct data can help improve the performance. The current training dataset has some noise.
Q4: I've experienced similar issues, typically caused by the Carla simulator crashing. When the "server" crashes, the "client" inevitably crashes as well. Unfortunately, resolving the simulator's crashes can be quite challenging. My approach is to monitor the system's running status and restart the data collection process regularly when needed.