Scene Graph Generation in Autonomous Driving: a neuro-symbolic approach

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contact
5. Acknowledgments

About The Project

The master thesis explores the usage of a Neuro-Symbolic Relational TRansformer (Nesy-RelTR) in the context of Visual Relationship Detection for Autonomous Driving scenarios.

Distributed under the Creative Commons Attribution Non-commercial No Derivatives.

If you find this project useful in your research, please consider cite:

@mastersthesis{sgg_av,
  author    = {Paolo E.I. Dimasi},
  title     = {Scene Graph Generation in Autonomous Driving: a Neuro-symbolic Approach},
  school    = {Politecnico di Torino},
  year      = {2023},
  address   = {Turin, Italy},
  url       = {http://webthesis.biblio.polito.it/id/eprint/29354}
}

(back to top)

Getting Started

Prerequisites

Python 3.10+

Installation

1. Clone the repo

   git clone https://github.com/Pamasi/sgg_av.git

2. Create an Anaconda environment

   conda create -n sgg_av_env python=3.10.12 
   conda activate sgg_av_env
   python -m pip install -r requirements.txt

3. Generate Traffic Genome dataset (folder coco_traffic_genome_v2) and its extension with Visual Genome (folder coco_mix_dataset_v2)

   . generate_tg.sh 

WARNING

Torchviz is not compatible with RelTR do not install it.

Create RDF Knowledge graph from pandaset

  python generate_kg/pandaset2RDF.py -f <pandaset-metadata-folder> -d <dataset-name> 

(back to top)

Usage

Knowledge Graph Embedding Generation

  python train_kge.py

RelTR: KGE

  python train_reltr.py --enable_kge --kge_path <kge_ckpt_path>

RelTR: LTN

  python train_reltr.py --enable_ltn

Inference

  python inference.py  --in_xywh --resume $model_path --img_path <image_path> --conf <confidence_score> --inf_dir <output-folder> 

Contact

Email: paolo.dimasi@outlook.com

(back to top)

Acknowledgments

• RelTR: Relation Transformer for Scene Graph Generation
• Traffic Genome Dataset
• LTNtorch
• nuScenes Knowledge Graph - A comprehensive semantic representation of traffic scenes for trajectory prediction
• pandaset-devkit

(back to top)