DEFT-Detection-Embeddings-for-Tracking.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DEFT-Detection-Embeddings-for-Tracking.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO8FI9isrkekpWade+T6hj9",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hardik0/Multi-Object-Tracking-Google-Colab/blob/main/DEFT-Detection-Embeddings-for-Tracking.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aAGc6cbIza35"
      },
      "source": [
        "# DEFT: Detection Embeddings for Tracking\r\n",
        "\r\n",
        "<img src=\"https://github.com/MedChaabane/DEFT/raw/main/DEFT_Diagram.png\" height=300px>\r\n",
        "\r\n",
        "**Abstract**\r\n",
        "\r\n",
        "In this paper, we propose an efficient joint detection and tracking model named DEFT, or Detection Embeddings for Tracking. Our approach relies on an appearance-based object matching network jointly-learned with an underlying object detection network. An LSTM is also added to capture motion constraints. DEFT has comparable accuracy and speed to the top methods on 2D online tracking leaderboards while having significant advantages in robustness when applied to more challenging tracking data. DEFT raises the bar on the nuScenes monocular 3D tracking challenge, more than doubling the performance of the previous top method.\r\n",
        "\r\n",
        "**Original code :** [DEFT: Detection Embeddings for Tracking](https://github.com/MedChaabane/DEFT) by\r\n",
        "[Mohamed Chaabane](https://github.com/MedChaabane), Peter Zhang, J. Ross Beveridge, Stephen O'Hara\r\n",
        "```\r\n",
        "@article{Chaabane2021deft,\r\n",
        "  title={DEFT: Detection Embeddings for Tracking},\r\n",
        "  author={Chaabane, Mohamed and Zhang, Peter and Beveridge, Ross and O'Hara, Stephen},\r\n",
        "  journal={arXiv preprint arXiv:2102.02267},\r\n",
        "  year={2021}\r\n",
        "}\r\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "etGg4gKDLBte"
      },
      "source": [
        "! nvidia-smi\r\n",
        "! /usr/local/cuda/bin/nvcc --version"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CLmcrg3LLrwt"
      },
      "source": [
        "## Installation\r\n",
        "Clone this repo, and run the following commands."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HFZHLTy9LJGC"
      },
      "source": [
        "! git clone https://github.com/MedChaabane/DEFT\r\n",
        "%cd DEFT"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "srjNVoqyL7ht"
      },
      "source": [
        "Install PyTorch and the dependencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AcWB_59nL1Ad"
      },
      "source": [
        "! pip install torch==1.2.0 torchvision==0.4.0    # CUDA 10.0\r\n",
        "# Install COCOAPI:\r\n",
        "! pip install cython; pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'\r\n",
        "! pip install -r requirements.txt  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kB5Hq4nVCDC"
      },
      "source": [
        "Restart the runtime "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZFAN_zUSE4Le",
        "outputId": "0fb05e8c-5627-413b-ff68-196e3fa6ac78"
      },
      "source": [
        "%cd DEFT/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/DEFT\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5xhamqWSMrkf"
      },
      "source": [
        "Compile [DCNv2](https://github.com/CharlesShang/DCNv2)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pjFlGHCAMqr2"
      },
      "source": [
        "%%shell\r\n",
        "\r\n",
        "cd src/lib/model/networks/\r\n",
        "git clone https://github.com/CharlesShang/DCNv2\r\n",
        "cd DCNv2\r\n",
        "./make.sh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cx2zyTuZNNiB"
      },
      "source": [
        "Download the [pretrained models](https://drive.google.com/drive/folders/1dlVoV-4fMYlttdj2ba0unn6WX-nxaC48?usp=sharing) and move them to src/models/ ."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gHuNSj7GNOFN"
      },
      "source": [
        "#! pip install gdown\r\n",
        "%mkdir src/models\r\n",
        "! gdown --id 1H7RX3M7wUA55RxeZDkloby9foGgYlhcp -O src/models/model_mot.pth\r\n",
        "#! gdown --id 1WFSaZmnT2-xacjf7L9nkudmbShERFzX_ -O src/models/model_kitti.pth"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PkcSS7tHPovc"
      },
      "source": [
        "## Datsets Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3fORFdvPudU"
      },
      "source": [
        "**MOT 2017**\r\n",
        "* Run the dataset preprocessing script."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ljmz4LU-PzMT"
      },
      "source": [
        "# update get_mot_17.sh - unzip -q MOT17.zip\r\n",
        "%%shell\r\n",
        "\r\n",
        "mkdir data\r\n",
        "cd src/tools/\r\n",
        "sh get_mot_17.sh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jk5N3cUYP39B"
      },
      "source": [
        "The output data structure should be:\r\n",
        "```\r\n",
        "    ${DEFT_ROOT}\r\n",
        "  |-- data\r\n",
        "  `-- |-- mot17\r\n",
        "      `-- |--- train\r\n",
        "          |   |--- MOT17-02-FRCNN\r\n",
        "          |   |    |--- img1\r\n",
        "          |   |    |--- gt\r\n",
        "          |   |    |   |--- gt.txt\r\n",
        "          |   |    |--- det\r\n",
        "          |   |    |   |--- det.txt\r\n",
        "          |   |--- ...\r\n",
        "          |--- test\r\n",
        "          |   |--- MOT17-01-FRCNN\r\n",
        "          |---|--- ...\r\n",
        "          `---| annotations\r\n",
        "              |--- train.json\r\n",
        "              `--- test.json\r\n",
        "```\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_EGa8q7zAUv"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EVJR4yEZiLHp",
        "outputId": "a309fc49-cc83-48af-b47a-43640d095450"
      },
      "source": [
        "cd src"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/DEFT/src\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qQn5SGbCiMpT",
        "outputId": "55fa14e8-714b-4a0e-9d71-b057a2f44010"
      },
      "source": [
        "# edit train.py \r\n",
        "#import warnings\r\n",
        "#warnings.filterwarnings(\"ignore\", category=UserWarning)\r\n",
        "\r\n",
        "! python train.py tracking --exp_id mot17_train --dataset mot --dataset_version 17trainval --pre_hm --ltrb_amodal --same_aug --hm_disturb 0.05 --lost_disturb 0.4 --fp_disturb 0.1 --gpus 0 --load_model models/model_mot.pth"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Running tracking\n",
            "Using tracking threshold for out threshold! 0.3\n",
            "Fix size testing.\n",
            "training chunk_sizes: [4]\n",
            "Namespace(AFE=True, K=100, add_05=False, amodel_offset_weight=1, arch='dla_34', aug_rot=0, backbone='dla34', batch_size=4, chunk_sizes=[4], custom_dataset_ann_path='', custom_dataset_img_path='', data_dir='/content/DEFT/src/lib/../../data', dataset='mot', dataset_version='17trainval', debug=0, debug_dir='/content/DEFT/src/lib/../../exp/tracking/mot17_train/debug', debugger_theme='white', dense_reg=1, dep_weight=1, depth_scale=1, dim_weight=1, dla_node='dcn', down_ratio=4, efficient_level=0, eval_val=False, exp_dir='/content/DEFT/src/lib/../../exp/tracking', exp_id='mot17_train', fix_res=True, fix_short=-1, flip=0.5, flip_test=False, fp_disturb=0.1, gpus=[0], gpus_str='0', head_conv={'hm': [256], 'reg': [256], 'wh': [256], 'tracking': [256], 'ltrb_amodal': [256]}, head_kernel=3, heads={'hm': 1, 'reg': 2, 'wh': 2, 'tracking': 2, 'ltrb_amodal': 4}, hm_disturb=0.05, hm_hp_weight=1, hm_weight=1, hp_weight=1, hungarian=False, ignore_loaded_cats=[], input_h=544, input_res=960, input_w=960, keep_res=False, kitti_split='3dop', load_model='models/model_mot.pth', load_model_traj='', load_results='', lost_disturb=0.4, lr=0.000125, lr_step=[60], lstm=False, ltrb=False, ltrb_amodal=True, ltrb_amodal_weight=0.1, ltrb_weight=0.1, map_argoverse_id=False, master_batch_size=4, matching_weight=1, max_age=-1, max_frame_dist=1, max_frame_dist_AFE=10, max_object=100, model_output_list=False, msra_outchannel=256, neck='dlaup', new_thresh=0.3, nms=False, no_color_aug=False, no_pre_img=False, non_block_test=False, not_cuda_benchmark=False, not_idaup=False, not_max_crop=False, not_prefetch_test=False, not_rand_crop=False, not_set_cuda_env=False, not_show_bbox=False, not_show_number=False, not_show_txt=False, num_classes=1, num_epochs=80, num_head_conv=1, num_iters=-1, num_layers=101, num_stacks=1, num_workers=8, nuscenes_att=False, nuscenes_att_weight=1, off_weight=1, only_show_dots=False, optim='adam', out_thresh=0.3, output_h=136, output_res=240, output_w=240, pad=31, pre_hm=True, pre_img=True, pre_thresh=0.3, print_iter=0, prior_bias=-4.6, public_det=False, qualitative=False, reg_loss='l1', reset_hm=False, resize_video=False, resume=False, reuse_hm=False, root_dir='/content/DEFT/src/lib/../..', rot_weight=1, rotate=0, same_aug_pre=True, save_all=False, save_dir='/content/DEFT/src/lib/../../exp/tracking/mot17_train', save_framerate=30, save_img_suffix='', save_imgs=[], save_point=[90], save_results=False, save_video=False, scale=0, seed=317, shift=0, show_trace=False, show_track_color=False, skip_first=-1, tango_color=False, task='tracking', test=False, test_dataset='mot', test_focal_length=-1, test_scales=[1.0], track_buffer=30, track_thresh=0.3, tracking=True, tracking_weight=1, trainval=False, transpose_video=False, use_kpt_center=False, use_loaded_results=False, val_intervals=10000, velocity=False, velocity_weight=1, video_h=512, video_w=512, vis_gt_bev='', vis_thresh=0.3, weights={'hm': 1, 'reg': 1, 'wh': 0.1, 'tracking': 1, 'ltrb_amodal': 0.1}, wh_weight=0.1, zero_pre_hm=False, zero_tracking=False)\n",
            "Creating model...\n",
            "Warning: No ImageNet pretrain!!\n",
            "loaded models/model_mot.pth, epoch 70\n",
            "Setting up train data...\n",
            "Using MOT 17 17trainval\n",
            "ann_file train.json\n",
            "==> initializing train data from /content/DEFT/src/lib/../../data/mot17/annotations/train.json, \n",
            " images from /content/DEFT/src/lib/../../data/mot17/train ...\n",
            "loading annotations into memory...\n",
            "Done (t=0.35s)\n",
            "creating index...\n",
            "index created!\n",
            "Creating video index!\n",
            "Loaded MOT 17trainval train 5316 samples\n",
            "Starting training...\n",
            "yesyes \n",
            "\u001b[Ktracking/mot17_train |################################| train: [1][1328/1329]|Tot: 1:24:24 |ETA: 0:00:04 |tot 0.1409 |hm 0.3169 |wh 0.7187 |reg 0.1742 |ltrb_amodal 0.7420 |tracking 0.0035 |Data 0.005s(0.010s) |Net 3.811s\n",
            "yesyes \n",
            "\u001b[Ktracking/mot17_train |########                        | train: [2][371/1329]|Tot: 0:23:41 |ETA: 1:00:42 |tot 0.1379 |hm 0.3263 |wh 0.7026 |reg 0.1744 |ltrb_amodal 0.7421 |tracking 0.0034 |Data 0.005s(0.024s) |Net 3.821s"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L5VqvzZNPO8A"
      },
      "source": [
        "%%shell \r\n",
        "\r\n",
        "cd src\r\n",
        "# train\r\n",
        "python train.py tracking --exp_id mot17_train --dataset mot --dataset_version 17trainval --pre_hm --ltrb_amodal --same_aug --hm_disturb 0.05 --lost_disturb 0.4 --fp_disturb 0.1 --gpus 0 --load_model models/model_mot.pth\r\n",
        "# train the motion model\r\n",
        "python train_prediction.py tracking --exp_id mot17_motion_model --dataset mot --dataset_version 17trainval --gpus 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ywdNkJoRXT2L"
      },
      "source": [
        "## Test & Visualization "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dn96B0hIK02m"
      },
      "source": [
        "%%shell\r\n",
        "\r\n",
        "cd src\r\n",
        "# test\r\n",
        "python test.py tracking --exp_id mot17_train --dataset mot --dataset_version 17test --pre_hm --ltrb_amodal --track_thresh 0.4 --pre_thresh 0.5 --load_model models/model_mot.pth"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gl3pLqPDXjbq"
      },
      "source": [
        "## Demo - To-Do list;"
      ]
    }
  ]
}