3Surrey Institute for People-Centred Artificial Intelligence, UK
Accepted to ECCV 2022
 |
- (June, 2022) We released TAGS training and inference code for ActivityNetv1.3 dataset.
- (June, 2022) TAGS is accepted by ECCV 2022.
- First proposal-free framework for Temporal Action Detection (TAD) task.
- Converted the traditional regression based Boundary Prediction into a classification problem.
- Predicts Global Masks for action instances and generates start/end points from mask start/end points.
- A novel Boundary Refinement and Alignment strategy is proposed which ensures inter-branch consistency.
- Efficient and Faster than all the existing Anchor-based and Anchor-Free approaches.
Regarding any issues on the implementation you can contact the first author at s.nag@surrey.ac.uk
Existing temporal action detection (TAD) methods rely on generating an overwhelmingly large number of proposals per video. This leads to complex model designs due to proposal generation and/or per-proposal action instance evaluation and the resultant high computational cost. In this work, for the first time, we propose a proposal-free Temporal Action detection model with Global Segmentation mask (TAGS). Our core idea is to learn a global segmentation mask of each action instance jointly at the full video length. The TAGS model differs significantly from the conventional proposal-based methods by focusing on global temporal representation learning to directly detect local start and end points of action instances without proposals. Further, by modeling TAD holistically rather than locally at the individual proposal level, TAGS needs a much simpler model architecture with lower computational cost. Extensive experiments show that despite its simpler design, TAGS outperforms existing TAD methods, achieving new state-of-the-art performance on two benchmarks. Importantly, it is ~ 20x faster to train and ~1.6x more efficient for inference.
- Python 3.7
- PyTorch == 1.9.0 (Please make sure your pytorch version is atleast 1.8)
- NVIDIA GPU
- Kornia
It is suggested to create a Conda environment and install the following requirements
pip3 install -r requirements.txtDownload the video features and update the Video paths/output paths in config/anet.yaml file. For now ActivityNetv1.3 dataset config is available. We are planning to release the code for THUMOS14 dataset soon. We obtain the I3D features from ACM-Net
and also process the annotations similar to TSP
| Dataset | Feature Backbone | Pre-Training | Link |
|---|---|---|---|
| ActivityNet | TSN | Kinetics-400 | Google Drive |
| THUMOS | TSN | Kinetics-400 | Google Drive |
| ActivityNet | I3D | Kinetics-400 | Google Drive |
| THUMOS | I3D | Kinetics-400 | Google Drive |
To train TAGS from scratch run the following command. The training configurations can be adjusted from config/anet.yaml file.
python tags_train.pyWe provide the pretrained models containing the checkpoint for I3D features on ActivityNetv1.3 . It can be found in the Link
After downloading the checkpoints, the checkpoints path can be saved in config/anet.yaml file.
The model inference can be then performed using the following command
python tags_inference.pyTo evaluate our TAGS model run the following command.
python eval.py
- Fix the multi-scale Transformer code in Snippet Embedding
- Support for THUMOS14 dataset
- Enable multi-gpu training
If you find this project useful for your research, please use the following BibTeX entry.
@article{nag2022temporal,
title={Temporal Action Detection with Global Segmentation Mask Learning},
author={Nag, Sauradip and Zhu, Xiatian and Song, Yi-Zhe and Xiang, Tao},
journal={arXiv preprint arXiv:2207.06580},
year={2022}
}
}