NLQ(*) and MQ(*) backbone checkpoints

[srama2512]

Hi,

Thank you for releasing the code, checkpoints, and features. I was unable to find the checkpoints corresponding to the NLQ and MQ verb/noun features. I'm interested in extracting features for videos outside the NLQ/MQ dataset. Can you share these checkpoints? It will also be great if you can share the script to extract features from videos. Thanks in advance.

[cg1177]

We have to spend some time to sort it out and release it before the Spring Festival.

[srama2512]

Is it possible to release only the checkpoints sooner in that case?

[cg1177]

The checkpoints you need have been released before. You can find them in the readme.

[srama2512]

@cg1177 - Thanks for your response. Is there a difference between the features provided for NLQ/MQ here and the checkpoints here? From the report, I got the impression that the backbones were finetuned for the tasks as well. Please correct me if I'm wrong.

[cg1177]

This repository provides the weights pretrained on the verb and noun subsets. The released features are extracted by these weights. We finetune the model for the MQ task again, but we have not released MQ features. In our report, only using verb features for the MQ task also achieves a nice performance. But for NLQ tasks, we did not perform the second finetuning.

[srama2512]

Great. Thank you for the clarification!

[srama2512]

@cg1177 - For feature extraction in NLQ and MQ, did you use the features after or before fc_norm?

[cg1177]

no

[srama2512]

As in, you used the features after fc_norm? I found it was necessary to use those to match the norms of pre-computed features.

[cg1177]

In practice, we did not use fc_norm of pretrained backbone. We did not study this setting. If you can comfirm that normalized features are better than the non-normalized, please tell me. We will also perform some experiments to verify it.

[srama2512]

@cg1177 - Hmm, the features released for NLQ are most likely using fc_norm though. I'm not suggesting that this is empirically better or worse than not using it, but it is essential for the consistency of feature generation for new videos. I regenerated verb features for the NLQ clip fae92e70-88aa-4b77-b41a-5879b74c804c with fc_norm enabled (and classifier FC manually removed). I sampled video frames at 30 fps (the raw video fps) and divided them into non-overlapping sets of 16 frames. For each set of 16 frames, I used the pre-processing suggested from this line. Here are some statistics for the released verb features (x_old) and the ones I computed using the released verb checkpoint (x_new).

In [1]: import torch

In [2]: x_new = torch.load("<NEW_FEATURES_DIR>/fae92e70-88aa-4b77-b41a-5879b74c804c.pt")

In [3]: x_old = torch.load("<RELEASED_FEATURES_DIR>/fae92e70-88aa-4b77-b41a-5879b74c804c.pt")

The features I created add one extra length (not sure why?).

In [4]: x_old.shape
Out[4]: torch.Size([899, 1024])

In [5]: x_new.shape
Out[5]: torch.Size([900, 1024])

But the vector norms are similar, and the cosine similarity is high for nearby features.

In [7]: x_new[0].norm()
Out[7]: tensor(14.7372)

In [8]: x_old[0].norm()
Out[8]: tensor(14.7915)

In [9]: torch.nn.functional.cosine_similarity(x_old[0], x_new[0], dim=0)
Out[9]: tensor(0.9706)

In [10]: torch.nn.functional.cosine_similarity(x_old[0], x_new[1], dim=0)
Out[10]: tensor(0.9357)

In [11]: torch.nn.functional.cosine_similarity(x_old[0], x_new[20], dim=0)
Out[11]: tensor(0.5504)

Here are the min/max statistics of vector norms over time.

In [12]: x_new.norm(dim=1).min()
Out[12]: tensor(14.4686)

In [13]: x_new.norm(dim=1).max()
Out[13]: tensor(18.2402)

In [14]: x_old.norm(dim=1).min()
Out[14]: tensor(14.7437)

In [15]: x_old.norm(dim=1).max()
Out[15]: tensor(18.2667)

[cg1177]

Hi. We confirmed and found that fc_norm is indeed used. The different sequence lengths should be because we did not padding the last window but discarded it. In addition, we also found that there are some different settings between the feature extractor and model testing. We will check further.