End-to-end Autonomous Driving

This repo is all you need for end-to-end autonomous driving research. We present awesome talks, comprehensive paper collections, benchmarks, and challenges.

Table of Contents

• At a Glance
• Learning Materials for Beginners
• Workshops and Talks
• Paper Collection
• Benchmarks and Datasets
• Competitions / Challenges
• Contributing
• License
• Citation
• Contact

At a Glance

The autonomous driving community has witnessed a rapid growth in approaches that embrace an end-to-end algorithm framework, utilizing raw sensor input to generate vehicle motion plans, instead of concentrating on individual tasks such as detection and motion prediction. In this survey, we provide a comprehensive analysis of more than 270 papers on the motivation, roadmap, methodology, challenges, and future trends in end-to-end autonomous driving. More details can be found in our survey paper.

End-to-end Autonomous Driving: Challenges and Frontiers

Li Chen^1,2, Penghao Wu¹, Kashyap Chitta^3,4, Bernhard Jaeger^3,4, Andreas Geiger^3,4, and Hongyang Li^1,2

¹ OpenDriveLab, Shanghai AI Lab, ² University of Hong Kong, ³ University of Tübingen, ⁴ Tübingen AI Center

If you find some useful related materials, shoot us an email or simply open a PR!

(back to top)

Learning Materials for Beginners

Online Courses

• Lecture: Self-Driving Cars, Andreas Geiger, University of Tübingen, Germany
• Self-Driving Cars Specialization, University of Toronto, Coursera
• The Complete Self-Driving Car Course - Applied Deep Learning, Udemy
• Self-Driving Car Engineer Nanodegree Program, Udacity

(back to top)

Workshops and Talks

Workshops (recent years)

• [CVPR 2024] Foundation Models for Autonomous Systems
• [CVPR 2024] Tutorial: End-to-End Autonomy: A New Era of Self-Driving
• [CVPR 2024] Tutorial: Towards Building AGI in Autonomy and Robotics
• [CVPR 2023] Workshop on End-to-end Autonomous Driving
• [CVPR 2023] End-to-End Autonomous Driving: Perception, Prediction, Planning and Simulation
• [ICRA 2023] Scalable Autonomous Driving

Workshops (previous years)

• [NeurIPS 2022] Machine Learning for Autonomous Driving
• [IROS 2022] Behavior-driven Autonomous Driving in Unstructured Environments
• [ICRA 2022] Fresh Perspectives on the Future of Autonomous Driving Workshop
• [NeurIPS 2021] Machine Learning for Autonomous Driving
• [NeurIPS 2020] Machine Learning for Autonomous Driving
• [CVPR 2020] Workshop on Scalability in Autonomous Driving

Talks

Relevant talks from other workshops

• Common Misconceptions in Autonomous Driving - Andreas Geiger, Workshop on Autonomous Driving, CVPR 2023
• Learning Robust Policies for Self-Driving - Andreas Geiger, AVVision: Autonomous Vehicle Vision Workshop, ECCV 2022
• Autonomous Driving: The Way Forward - Vladlen Koltun, Workshop on AI for Autonomous Driving, ICML 2020
• Feedback in Imitation Learning: Confusion on Causality and Covariate Shift - Sanjiban Choudhury and Arun Venkatraman, Workshop on AI for Autonomous Driving, ICML 2020

(back to top)

Paper Collection

We list key challenges from a wide span of candidate concerns, as well as trending methodologies. Please refer to this page for the full list, and the survey paper for detailed discussions.

• Survey
• Language / VLM for Driving
  • Review for VLM in Driving
  • Papers for VLM in Driving
• World Model & Model-based RL
• Multi-sensor Fusion
• Multi-task Learning
• Interpretability
  • Review for Interpretability
  • Attention Visualization
  • Interpretable Tasks
  • Cost Learning
  • Linguistic Explainability
  • Uncertainty Modeling
  • Counterfactual Explanations and Causal Inference
• Visual Abstraction / Representation Learning
• Policy Distillation
• Causal Confusion
• Robustness
  • Long-tailed Distribution
  • Covariate Shift
  • Domain Adaptation
• Affordance Learning
• BEV
• Transformer
• V2V Cooperative
• Distributed RL
• Data-driven Simulation
  • Parameter Initialization
  • Traffic Simulation
  • Sensor Simulation

(back to top)

Benchmarks and Datasets

Real-world deployment is the final benchmark for autonomous driving. However, testing in the real world is expensive. For academic benchmarking, we recommend you read this write-up from Jaeger et al. 2024: Common mistakes in benchmarking.

Closed-loop

• CARLA
  • Leaderboard 1.0
  • Leaderboard 2.0
• nuPlan
  • Leaderboard (inactive after the CVPR 2023 challege)
  • NAVSIM

Open-loop

• nuScenes
• nuPlan
• Argoverse
• Waymo Open Dataset

(back to top)

Competitions / Challenges

• End-to-End Driving at Scale, Foundation Models for Autonomous Systems, CVPR 2024
• CARLA Autonomous Driving Challenge, Foundation Models for Autonomous Systems, CVPR 2024
• nuPlan planning, Workshop on End-to-end Autonomous Driving, CVPR 2023
• CARLA Autonomous Driving Challenge 2022, Machine Learning for Autonomous Driving, NeurIPS 2022
• CARLA Autonomous Driving Challenge 2021, Machine Learning for Autonomous Driving, NeurIPS 2021
• CARLA Autonomous Driving Challenge 2020, Machine Learning for Autonomous Driving, NeurIPS 2020
• Learn-to-Race Autonomous Racing Virtual Challenge, 2022
• INDY Autonomous Challenge

(back to top)

Contributing

Thank you for all your contributions. Please make sure to read the contributing guide before you make a pull request.

(back to top)

License

End-to-end Autonomous Driving is released under the MIT license.

(back to top)

Citation

If you find this project useful in your research, please consider citing:

@article{chen2023e2esurvey,
  title={End-to-end Autonomous Driving: Challenges and Frontiers},
  author={Chen, Li and Wu, Penghao and Chitta, Kashyap and Jaeger, Bernhard and Geiger, Andreas and Li, Hongyang},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2024}
}

(back to top)

Contact

Primary contact: hy@opendrivelab.com. You can also contact: lichen@opendrivelab.com.

Join OpenDriveLab Slack to chat with the commuty! Slack channel: #e2ead.

(back to top)