This repository hosts code and models of the paper Deep Learning-Based Object Tracking via Compressed Domain Residual Frames by Karim El Khoury, Jonathan Samelson and Benoît Macq.
Our goal in using residual frames to perform object tracking is to address three major needs in video surveillance:
- The need for video compression
- The need for video analysis
- The need for privacy protection
For this purpose, the residual frame is a light representation that comes from the compressed domain at no additional cost. Our results show that using this representation can be just as effective as using classical decoded frames while reducing the amount of information leakage in a video stream.
For instance, residual frames can be used to detect only objects of interest or to reduce the number of false positives as they only feature moving objects.
It is structured as follows:
/src/ABMA/contains the Matlab code of the Adaptive Block Matching Algorithm (ABMA) allows to generate the residual frame of a video sequence. See ABMA Readme for specific instructions./src/inferencecontains the Python code to obtain the tracking results (in MOT format) using decoded frames (classical representation) or the residual frames (representation from the compressed domain). See Inference Readme for specific instructions.
Once the results are obtained, HOTA metric can be used to evaluate the performance of the tracking. See TrackEval for the instructions. Note that in our article we used the repository dated 7th April 2021.
The YOLOv4 and Tiny-YOLOv4 weights and config files used to get the detections can be downloaded via our Google Drive. Those models are trained to detect vehicles. The models to detect objects in decoded images were trained on MIO-TCD Localization dataset while the models to detect objects in residuals frames were trained on a custom dataset based on AICity 2021 Track 1 & 3, whose the access can only be granted by AICity. Please see our article for more information about our models.
Refer to the following repositories for more information on individual algorithms. Part of the code present in this repository was adapted from those repositories, see individual source files for details.
YOLO & Tiny-YOLO training: https://github.com/AlexeyAB/darknet
YOLO & Tiny-YOLO inference: OpenCV Documentation - DNN module
SORT: https://github.com/abewley/sort
KIOU: https://github.com/siyuanc2/kiout
Block Matching Algorithm for Motion Estimation: Block Matching Algorithms for Motion Estimation
This repository is released under GPL-3.0 License, please consult LICENSE file for details.