CARLA RAI Challenge 2024 Starter kit

This repository contains the starter kit for the CARLA RAI Challenge 2024. It is based on Neural Attention Fields for End-to-End Autonomous Driving (NEAT).

Neural Attention Fields for End-to-End Autonomous Driving (NEAT)

Paper | Supplementary | Video | Talk | Poster | Slides

Clone

To clone the repository with all submodules

git clone --recurse-submodules https://github.com/cognitive-robots/rai-neat-starter-kit.git 

Setup

Please follow the installation instructions from our TransFuser repository to set up the CARLA simulator. The conda environment required for NEAT can be installed via:

conda env create -f environment.yml
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

Data Generation (from NEAT)

The training data is generated using team_code/auto_pilot.py. Data generation requires routes and scenarios. Each route is defined by a sequence of waypoints (and optionally a weather condition) that the agent needs to follow. Each scenario is defined by a trigger transform (location and orientation) and other actors present in that scenario (optional). We provide several routes and scenarios under leaderboard/data/. The TransFuser repository and leaderboard repository provide additional routes and scenario files.

Running a CARLA Server

With Display

Without Docker:

./CarlaUE4.sh

With Docker:

Instructions for setting up docker are available here. The following command will pull the docker image of CARLA 0.9.10.1 and run the container.

docker run --privileged --gpus all --net=host -e DISPLAY=${DISPLAY} -it -e SDL_VIDEODRIVER=x11 -v /tmp/.X11-unix:/tmp/.X11-unix carlasim/carla:0.9.10.1 /bin/bash

Once inside the container, run the Carla server with

./CarlaUE4.sh

Running the Autopilot

Once the CARLA server is running, rollout the autopilot to start data generation.

bash rai/scripts/run_evaluation.sh

The expert agent used for data generation is defined in team_code/auto_pilot.py. Different variables which need to be set are specified in rai/scripts/run_evaluation.sh. The expert agent is originally based on the autopilot from this codebase.

Training (from NEAT)

The training code and pretrained models are provided below.

mkdir model_ckpt
wget https://s3.eu-central-1.amazonaws.com/avg-projects/neat/models.zip -P model_ckpt
unzip model_ckpt/models.zip -d model_ckpt/
rm model_ckpt/models.zip

There are 5 pretrained models provided in model_ckpt/:

• AIM-MT (2D): aim_mt_sem and aim_mt_sem_depth
• AIM-MT (BEV): aim_mt_bev
• AIM-VA: aim_va
• NEAT: neat

Additional baselines are available in the TransFuser repository.

Evaluation and Quick Test

Running CARLA Server

With Docker:

Instructions for setting up docker are available here. The following command will pull the docker image of CARLA 0.9.10.1 and run the container.

docker run --privileged --gpus all --net=host -e DISPLAY=${DISPLAY} -it -e SDL_VIDEODRIVER=x11 -v /tmp/.X11-unix:/tmp/.X11-unix carlasim/carla:0.9.10.1 /bin/bash

Once inside the container, run the Carla server with

./CarlaUE4.sh

Running NEAT Agent

Update the required variables in rai/scripts/env_var.sh and run the agent, with or without Docker.

Note: For a quick test, change CUSTOM_ROUTE_TIMEOUT (e.g. =15) in rai/scripts/env_var.sh.

Without Docker

bash rai/scripts/run_evaluation.sh

With Docker

Build the Docker image by running:

bash rai/scripts/make_docker.sh -t <image:tag> # for example neat:0.1

Once the image is built, run the Docker image with:

docker run --ipc=host --gpus all --net=host -e DISPLAY=$DISPLAY -it -e SDL_VIDEODRIVER=x11 -v /tmp/.X11-unix:/tmp/.X11-unix <image:tag> /bin/bash

Finally, run the evaluation script within the Docker container:

bash rai/scripts/run_evaluation.sh

Submission

To submit an agent to the CARLA RAI Challenge 2024:

1. First, create a Conda environment and install EvalAI, which is EvalAI's command-line interface package:

   conda create -n evalai-cli python=3.7
   conda activate evalai-cli
   pip install evalai~=1.3.18

2. If the Conda environment already exists, update the Conda environment path in rai/scripts/submit_evalai.sh and execute the script. This will activate your Conda environment and submit the agent:

   bash rai/scripts/submit_evalai.sh -t <image:tag> # for example neat:0.1