DeFlow: Self-supervised 3D Motion Estimation of Debris Flow ⛰️

Liyuan Zhu, Yuru Jia, Shengyu Huang, Nicholas Meyer, Andreas Wieser, Konrad Schindler, Jordan Aaron

ETH Zurich

Paper | Website | Dataset

This repository is the official implementation of paper: DeFlow: Self-supervised 3D Motion Estimation of Debris Flow, CVPRW 2023.

Existing work on scene flow estimation focuses on autonomous driving and mobile robotics, while automated solutions are lacking for motion in nature, such as that exhibited by debris flows. We propose \deflow, a model for 3D motion estimation of debris flows, together with a newly captured dataset. We adopt a novel multi-level sensor fusion architecture and self-supervision to incorporate the inductive biases of the scene. We further adopt a multi-frame temporal processing module to enable flow speed estimation over time. Our model achieves state-of-the-art optical flow and depth estimation on our dataset, and fully automates the motion estimation for debris flows.

Installation 🏞️

First clone our repository:

git clone https://github.com/Zhu-Liyuan/DeFlow
cd DeFlow

You will need to install conda to build the environment.

conda create -n DeFlow python=3.9
conda activate DeFlow
pip install -r requirements.txt

Dataset and pretrained model

We provide preprocessed debris flow dataset. The preprocessed dataset and checkpoint can be downloaded by running:

wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/DeFlow_Dataset.zip
unzip DeFlow_Dataset.zip -d data
wget --no-check-certificate --show-progress https://share.phys.ethz.ch/~gsg/DeFlow/checkpoint.zip
unzip checkpoint.zip

You can also build your own dataset following the structure below

├── Data
    ├── Cam1
        ├── 000001.jpg
        ├── 000002.jpg
        .
        .
        ├── 00000X.jpg
    ├── Cam2
        ├── 000001.jpg
        ├── 000002.jpg
        .
        .
        ├── 00000X.jpg
    ├── LiDAR
        ├── 000001.ply
        ├── 000002.ply
        .
        .
        ├── 00000X.ply
├── Transformations
        ├── cam_intrinxics.txt
        ├── LiCam_tranformations.txt

To train a model, run:

python main.py --config_path configs/deflow_default.yaml

and you can change the mode to eval in the config file for evaluation.

Contact

If you have any questions, please let me know:

• Liyuan Zhu {liyzhu@student.ethz.ch}

Citation

If you use DeFlow for any academic work, please cite our original paper.

@InProceedings{zhu2023DeFlow,
author = {Liyuan Zhu and Yuru Jia and Shengyu Huang and Nicholas Meyer and Andreas Wieser and Konrad Schindler, Jordan Aaron},
title = {DEFLOW: Self-supervised 3D Motion Estimation of Debris Flow},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023}
}

Additionally, we thank the respective developers of the following open-source projects:

• PCAccumulation
• CamLiFlow
• Self-mono-sf