MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval

This is an official implementation of MUSE built on model CLIP4clip.

Requirement

# Pytorch version
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
# From CLIP4clip
pip install ftfy regex tqdm
pip install opencv-python boto3 requests pandas

Install Causal_conv1d and Mamba_ssm following Vim.

Data Preparing

For MSRVTT

The official data and video links can be found in link.

For the convenience, you can also download the splits and captions by,

wget https://github.com/ArrowLuo/CLIP4Clip/releases/download/v0.0/msrvtt_data.zip

Besides, the raw videos can be found in sharing from Frozen️ in Time, i.e.,

wget https://www.robots.ox.ac.uk/~maxbain/frozen-in-time/data/MSRVTT.zip

Train MSR-VTT

bash train_msrvtt.sh

Acknowledgments

Our code is based on CLIP, CLIP4clip and Vim.