Cannot reproduce the ReID accuracy of the paper

[WinKawaks]

Hi, thank you for the great work!

I trained this model in joint mode for all animal categories. And print the ReID Top-1 Accuracy of cat every 20 epoches. Here are some results:

cat_20.csv 28.521374744700328
cat_40.csv 29.5117613916473
cat_60.csv 30.463412916402564
cat_80.csv 30.486301852243113
cat_100.csv 30.28646383548137

I notice the performance of cat ReID in Table. 2 of your paper is 70.30. There is a huge gap between our results. How many epoches did you train? And are there any tricks to select the best checkpoint?

[WinKawaks]

I test the AUC on cat with the 100 epoches checkpoint and the result is 98.28297794209323. It's even higher than the performance of Table. 3 (98.04). Is there anything wrong with the ReID test code?

[mapooon]

Hi,
Thank you for reporting the issue.

And are there any tricks to select the best checkpoint?

We do not use any tricks to select the checkpoint. We just used the models trained for 10 epochs.

Is there anything wrong with the ReID test code?

We have been informed by at least one third party who was able to reproduce the results including cat's re-id.
Therefore, the issue might be related to your specific code, model, or environment.
You can start by trying our pre-trained models directly on our code without any modifications, instead of using your trained models.

[WinKawaks]

@mapooon Hi! I found the problem. I used DDP with 4 GPUs to train the model before and found the performance is very poor. And then I used just 1 GPU to train the model, the performance is on par with the results in the paper.
Then I realized that the PartialFC is different between different ranks. But you only save the model of rank_0.

[mapooon]

@WinKawaks
Hi,
Thank you for reporting the issue!
Indeed, we always used a single GPU for the experiments, so we did not encounter the problem.
If you use multiple GPUs, the following command should work correctly:
python -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=12581 src/train_arcface.py src/configs/cat.py --output outputs/cat/arcface
where --nproc_per_node represents the number of GPUs on each node.