This is the official repository for ECCV 2024 Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance (LoRAT).
[Models] [Raw Results] [Poster]
Assuming you have a working python environment with pip installed.
apt update
apt install -y libturbojpegCan be skipped if using NGC container. PyTorch version should be >= 2.0.
pip install torch torchvisionpip install -r requirements.txtThis codebase should also work on Windows and macOS for debugging purposes.
The paths should be organized as follows:
LaSOT
├── airplane
├── basketball
...
├── training_set.txt
└── testing_set.txt
LaSOT_Extension
├── atv
├── badminton
...
└── wingsuit
GOT-10k
├── train
│ ├── GOT-10k_Train_000001
│ ...
├── val
│ ├── GOT-10k_Val_000001
│ ...
└── test
├── GOT-10k_Test_000001
...
TrackingNet
├── TEST
├── TRAIN_0
...
└── TRAIN_11
COCO
├── annotations
│ ├── instances_train2017.json
│ └── instances_val2017.json
└── images
├── train2017
│ ├── 000000000009.jpg
│ ├── 000000000025.jpg
│ ...
└── val2017
├── 000000000139.jpg
├── 000000000285.jpg
...
TNL2K_TEST
├── advSamp_Baseball_game_002-Done
├── advSamp_Baseball_video_01-Done
...
Copy consts.template.yaml as consts.yaml and fill in the paths.
LaSOT_PATH: '/path/to/lasot'
LaSOT_Extension_PATH: '/path/to/lasot_ext'
GOT10k_PATH: '/path/to/got10k'
TrackingNet_PATH: '/path/to/trackingnet'
COCO_2017_PATH: '/path/to/coco2017'
TNL2K_TEST_PATH: '/path/to/tnl2k_test'Register an account at wandb, then login with the command:
wandb loginNote: Our code performs evaluation automatically when model training is complete.
- Model weight is saved in
/path/to/output/run_id/checkpoint/epoch_{last}/model.bin. - Performance metrics can be found on terminal output and wandb dashboard.
- Tracking results are saved in
/path/to/output/run_id/eval/epoch_{last}/.
# Train and evaluate LoRAT-B-224 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output
# Train and evaluate LoRAT-L-224 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin large
# Train and evaluate LoRAT-g-378 model on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin giant_378
# Train and evaluate LoRAT-L-224 model following GOT-10k protocol on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin large --mixin got10k
# Train and evaluate on specific GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --device_ids 0,1,2,3
# Train and evaluate on multiple nodes
./run.sh LoRAT dinov2 --output_dir /path/to/output --nnodes $num_nodes --node_rank $node_rank --master_address $master_node_ip --date 2024.03.07-04.59.08-976343You can set the default settings, e.g. output_dir, in run.sh.
# Train and evaluate LoRAT-B-224 model on single GPU
python main.py LoRAT dinov2 --output_dir /path/to/output
# Train and evaluate LoRAT-B-224 model on CPU
python main.py LoRAT dinov2 --output_dir /path/to/output --device cpu
# Train and evaluate LoRAT-B-224 model on all GPUs
python main.py LoRAT dinov2 --distributed_nproc_per_node $num_gpus --distributed_do_spawn_workers --output_dir /path/to/output
# Train and evaluate LoRAT-B-224 model on multiple nodes, run_id need to be set manually
python main.py LoRAT dinov2 --master_address $master_address --distributed_node_rank $node_rank distributed_nnodes $num_nodes --distributed_nproc_per_node $num_gpus --distributed_do_spawn_workers --output_dir /path/to/output --run_id $run_idSee python main.py --help for more options.
Note: If you encounter any issues with torch.compile, disable is with --mixin disable_torch_compile.
Note: You can disable wandb logging with --disable_wandb.
Our code performs evaluation automatically when model training is complete. You can run evaluation only with the following command:
# evaluation only, on all datasets, defined in config/_dataset/test.yaml
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin evaluation --weight_path /path/to/weight.binThe evaluated datasets are defined in config/_dataset/test.yaml.
Note that, as defined in config/LoRAT/run.yaml, we evaluate LaSOT Extension dataset three times. The final performance is the average of the three evaluations.
Once the full evaluation is done, result files are saved in /path/to/output/run_id/eval/epoch_{last}/TrackingNet-test.zip.
Submit this file to the TrackingNet evaluation server to get the result of TrackingNet test split.
# Train and evaluate LoRAT-B-224 model following GOT-10k protocol on all GPUs
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin got10kSubmit /path/to/output/run_id/eval/epoch_{last}/GOT10k-test.zip to the GOT-10k evaluation server to get the result of GOT-10k test split.
Evaluation only:
# evaluation only, on GOT-10k dataset
./run.sh LoRAT dinov2 --output_dir /path/to/output --mixin got10k --mixin evaluation --weight_path /path/to/weight.binNote that, as defined in config/LoRAT/_mixin/got10k.yaml, we evaluate GOT-10k dataset three times.
pip install vot-toolkitprepare the VOT dataset by running the following command:
cd /path/to/vot_workspace
vot initialize vot_stack(vots2024/main|tests/multiobject)fill the path to the VOT dataset in consts.yaml
VOTS2023_PATH: '/path/to/vots2023_workspace/sequences'
VOT_TESTS_MULTIOBJECT_PATH: '/path/to/vot_tests_workspace/sequences'# Run VOT experiment (vots2024/main stack) on LoRAT-g-378 with SAM-H segmentation model
python vot_main.py vots2024/main LoRAT dinov2 /path/to/output --mixin giant_378 --mixin segmentify_sam_h --tracker_name LoRAT --weight_path /path/to/lorat_model_weight.binThis page describes how to create a custom dataset for training and evaluation.
@inproceedings{lorat,
title={Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance},
author={Lin, Liting and Fan, Heng and Zhang, Zhipeng and Wang, Yaowei and Xu, Yong and Ling, Haibin},
booktitle={ECCV},
year={2024}
}