Collection of papers and other resources for object detection and tracking using deep learning

Static Detection

• Region Proposal
  • Scalable Object Detection Using Deep Neural Networks cvpr14 (pdf, notes)
  • Selective Search for Object Recognition ijcv2013 (pdf, notes)
• RCNN
  • Faster R-CNN Towards Real-Time Object Detection with Region Proposal Networks tpami17 (pdf, notes)
  • Mask R-CNN (pdf, (notes, arxiv, code (keras), code (tensorflow)) [Facebook AI Research]
• YOLO
  • You Only Look Once Unified, Real-Time Object Detection ax1605 (pdf, notes)
  • YOLO9000 Better, Faster, Stronger ax16_12 (pdf, notes)
  • YOLOv3 An Incremental Improvement ax180408 (pdf, notes)
• SSD
  • SSD Single Shot MultiBox Detector eccv16_ax16_12 (pdf, notes)
  • DSSD Deconvolutional Single Shot Detector ax1701.06659 (pdf, notes)
• RetinaNet
  • Feature Pyramid Networks for Object Detection ax170419 (pdf, notes)
  • Focal Loss for Dense Object Detection ax180207 (pdf, notes)
• Misc
  • OverFeat Integrated Recognition, Localization and Detection using Convolutional Networks ax1402 iclr14 (pdf, notes)
  • LSDA Large scale detection through adaptation nips14 ax14_11 (pdf, notes)

Video Detection

• Tubelet
  • Object Detection from Video Tubelets with Convolutional Neural Networks CVPR16 (pdf, notes)
  • Object Detection in Videos with Tubelet Proposal Networks ax1704 cvpr17 (pdf, notes)
• FGFA
  • Deep Feature Flow for Video Recognition (pdf, arxiv, code) [Microsoft Research]
  • Flow-Guided Feature Aggregation for Video Object Detection ax1708 iccv17 (pdf, notes)
  • Towards High Performance Video Object Detection ax171130 microsoft (pdf, notes)
• RNN
  • Online Video Object Detection using Association LSTM iccv17 (pdf, notes)
  • Context Matters Refining Object Detection in Video with Recurrent Neural Networks bmvc16 (pdf, notes)

Multi Object Tracking

• Learning to Track: Online Multi-object Tracking by Decision Making (ICCV 2015) (Stanford) (pdf, code (Matlab), project page, notes)
• Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependencies (arxiv April 2017) (Stanford) (pdf, arxiv, project page, notes)
• Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor (ICCV 2015) (NEC Labs) (pdf, author page, notes)
• A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects (highest MT on MOT2015) (University of Freiburg, Germany) (pdf, arxiv, author page, notes)

Single Object Tracking

• Deep Reinforcement Learning for Visual Object Tracking in Videos (arxiv April 2017) (USC-Santa Barbara, Samsung Research) (pdf, arxiv, author page, notes)
• Visual Tracking by Reinforced Decision Making (arxiv February 2017) (Seoul National University, Chung-Ang University) (pdf, arxiv, author page, notes)
• Action-Decision Networks for Visual Tracking with Deep Reinforcement Learning (CVPR 2017) (Seoul National University) (pdf, supplementary, project page, notes)
• End-to-end Active Object Tracking via Reinforcement Learning (arxiv 30 May 2017) (Peking University, Tencent AI Lab) (pdf, arxiv)

Deep Learning

• Do Deep Nets Really Need to be Deep (NIPS 2014) (pdf, notes)
• Synthetic Gradients
  • Decoupled Neural Interfaces using Synthetic Gradients (arxiv August 2016) (pdf, notes)
  • Understanding Synthetic Gradients and Decoupled Neural Interfaces (arxiv March 2017) (pdf, notes)

Unsupervised Learning

• Learning Features by Watching Objects Move (CVPR 2017) (pdf, notes)

Interpolation

• Video Frame Interpolation via Adaptive Convolution (CVPR 2017 / ICCV 2017) (pdf (cvpr17), (pdf (iccv17), ppt)

Datasets

• IDOT dataset
• UA-DETRAC Benchmark Suite
• GRAM Road-Traffic Monitoring
• Stanford Drone Dataset
• Ko-PER Intersection Dataset
• TRANCOS Dataset
• Urban Tracker Dataset
• DARPA VIVID / PETS 2005 dataset (Non stationary camera)
• KIT-AKS Dataset (No ground truth)
• CBCL StreetScenes Challenge Framework (No top down viewpoint)
• MOT 2015 (mostly street level camera viewpoint)
• MOT 2016 (mostly street level camera viewpoint)
• MOT 2017 (mostly street level camera viewpoint)
• PETS 2009 (No vehicles)
• PETS 2017 (Low density; mostly pedestrians)
• KITTI Tracking Dataset (No top down viewpoint; non stationary camera)

Lists

• Datasets
  • List of traffic surveillance datasets
• Single Object Tracking
  • List of deep learning based tracking papers
  • List of single object trackers with results on OTB
• Multi Object Tracking
  • List of multi object tracking papers
• Misc
  • List of Matlab frameworks, libraries and software

Tutorials

• Static Detection
  • Deep Learning for Object Detection: A Comprehensive Review
  • Review of Deep Learning Algorithms for Object Detection
  • A Simple Guide to the Versions of the Inception Network
  • R-CNN, Fast R-CNN, Faster R-CNN, YOLO - Object Detection Algorithms
  • A gentle guide to deep learning object detection
  • The intuition behind RetinaNet
  • YOLO—You only look once, real time object detection explained
  • Understanding Feature Pyramid Networks for object detection (FPN)
  • Fast object detection with SqueezeDet on Keras
  • Region of interest pooling explained
• Video Detection
  • How Microsoft Does Video Object Detection - Unifying the Best Techniques in Video Object Detection Architectures in a Single Model
• Deep RL
  • Deep Reinforcement Learning: Pong from Pixels
  • Demystifying Deep Reinforcement Learning

Code

• Multi Object Tracking

  • Combined Image- and World-Space Tracking in Traffic Scenes [ICRA 2017] [C++]
  • Learning to Track: Online Multi-Object Tracking by Decision Making [ICCV 2015] [MATLAB]
  • Multiple Hypothesis Tracking Revisited [ICCV 2015] [highest MT on MOT2015 among open source trackers] [MATLAB]
  • Joint Tracking and Segmentation of Multiple Targets [CVPR 2015] [MATLAB]
  • High-Speed Tracking-by-Detection Without Using Image Information [AVSS 2017] [Python]
  • Continuous Energy Minimization for Multitarget Tracking [TPAMI 2014 / CVPR 2011 / ICCV 2011] [MATLAB]
  • Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning [CVPR 2014] [MATLAB] (project)
  • Discrete-Continuous Energy Minimization for Multi-Target Tracking [CVPR 2012] [MATLAB] (project)
  • Multiple target tracking based on undirected hierarchical relation hypergraph [CVPR 2014] [C++] (author)
  • Globally-optimal greedy algorithms for tracking a variable number of objects [CVPR 2011] [MATLAB] (author)
  • The way they move: Tracking multiple targets with similar appearance [ICCV 2013] [MATLAB]

• Single Object Tracking

  • A collection of common tracking algorithms (2003-2012)
  • Detect to Track and Track to Detect (ICCV 2017)[MATLAB]
  • DeepTracking: Seeing Beyond Seeing Using Recurrent Neural Networks (AAAI 2016)[Torch 7]
  • Hierarchical Convolutional Features for Visual Tracking (ICCV 2015)[Matlab]
  • Learning Multi-Domain Convolutional Neural Networks for Visual Tracking (Winner of The VOT2015 Challenge)[Matlab/MatConvNet]
  • RATM: Recurrent Attentive Tracking Model[Python]
  • Visual Tracking with Fully Convolutional Networks (ICCV 2015)[Matlab]
  • Fully-Convolutional Siamese Networks for Object Tracking[Tensor flow]
  • Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking (ECCV 2016)[MATLAB]
  • ECO: Efficient Convolution Operators for Tracking (CVPR 2017)[MATLAB]
  • End-to-end representation learning for Correlation Filter based tracking (CVPR 2017)[MATLAB]
  • ROLO : Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking (ISCAS 2017)[tensorfow]
  • Deep SORT : Simple Online Realtime Tracking with a Deep Association Metric (ICIP 2017)[python]

• Static Detection and Matching

  • Frameworks
    • Tensorflow object detection API
      • Only the two SSD nets can run at 12.5 FPS on one GTX 1080 TI (less accurate than YOLO 604x604). Next two models at 4-5 FPS (4-5% mAP better than YOLO). Best model < 1 FPS. Currently code only allow inference of 1 image at a time. Speed might improve by 2.5 times when they allow multiple image inference.
    • Detectron
  • SSD
    • SSD-Tensorflow [tensorflow]
    • SSD-Tensorflow (tf.estimator) [tensorflow]
    • SSD-Tensorflow (tf.slim) [tensorflow]
    • SSD-Keras [keras]
    • SSD-Pytorch [pytorch]
    • Enhanced SSD with Feature Fusion and Visual Reasoning [NCA18] [TensorFlow]
  • RCNN
    • PVANet: Lightweight Deep Neural Networks for Real-time Object Detection
    • Mask R-CNN TensorFlow [TensorFlow]
    • Mask R-CNN Keras [keras]
    • Light-head R-CNN [cvpr18] [TensorFlow]
    • Evolving Boxes for Fast Vehicle Detection [icme18] [Caffe/Python]
  • YOLO
    • Darknet: Convolutional Neural Networks [c/python]
    • YOLO9000: Better, Faster, Stronger - Real-Time Object Detection. 9000 classes! [c/python]
    • Darkflow [tensorflow]
    • Pytorch Yolov2 [pytorch]
  • RFCN
    • RFCN (author) [caffe/matlab]
    • RFCN-tensorflow [tensorflow]
  • Region Proposal
    • MCG : Multiscale Combinatorial Grouping - Object Proposals and Segmentation (project) [tpami16/cvpr14] [python]
    • COB : Convolutional Oriented Boundaries (project) [tpami18/eccv16] [matlab/caffe]
  • FPN
    • Feature Pyramid Networks for Object Detection [caffe/python]
  • Misc
    • R-FCN: Object Detection via Region-based Fully Convolutional Networks
    • Relation Networks for Object Detection [cvpr18] [MXNet]
    • DeNet: Scalable Real-time Object Detection with Directed Sparse Sampling [iccv17(poster)] [theano]
  • Matching
    • Matchnet
    • Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches

• Video Detection

  • TPN: Tubelet Proposal Network (CVPR 2017)[Python]
  • T-CNN: Tubelets with Convolution Neural Networks (CVPR 2016)[Python]
  • Flow-Guided Feature Aggregation for Video Object Detection (NIPS 2016 / ICCV 2017)[Python/CUDA]

• Optical Flow

  • Fast Optical Flow using Dense Inverse Search (DIS)
  • FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

• Deep RL

  • Asynchronous Methods for Deep Reinforcement Learning

• Misc

  • Deformable Convolutional Networks
  • RNNexp