Custom distance metric during testing

[weidenka]

The docu says that:

All losses, miners, and regularizers accept a distance argument.

Why do testers not have a distance argument? Wouldn't it make sense to evaluate on the same distance metric that was used for training? I'm not too much into metric learning (yet) so, I might misunderstand some conceptual decisions.

[KevinMusgrave]

I agree, you should be able to customize the distance for the testers and AccuracyCalculator. It's something I'd like to add in v1.0. In the meantime unfortunately I don't have an easy workaround.

Here are my thoughts on implementation:

Currently the k-nearest-neighbor computation is done using faiss, which has a limited number of distance metrics. (Right now AccuracyCalculator is hardcoded to use either normalized or unnormalized L2 distance.)

To use custom metrics like SNRDistance, I'll just add a helper function that computes the distance matrix and then torch.topk.

Then I'll add a knn_fn argument to AccuracyCalculator, which will accept any function that returns knn results. This will allow you to use preset functions (i.e. a faiss wrapper or a PML distance wrapper), or your own custom function for computing knn.

[weidenka]

The distance matrix you already get from and BaseDistance class, right? I'd reuse that one. Except that sounds fine to me. Thanks a lot for this package btw!

[KevinMusgrave]

The distance matrix you already get from and BaseDistance class, right? I'd reuse that one.

Yes

Thanks a lot for this package btw!

👍

[KevinMusgrave]

This is available in v1.0.0:

from pytorch_metric_learning.testers import GlobalEmbeddingSpaceTester
from pytorch_metric_learning.distances import SNRDistance
from pytorch_metric_learning.utils.inference import CustomKNN

knn_func = CustomKNN(SNRDistance())
ac = AccuracyCalculator(knn_func=knn_func)
tester = testers.GlobalEmbeddingSpaceTester(accuracy_calculator=ac)

Relevant docs:

• Accuracy Calculation
• FaissKNN
• CustomKNN

To install:

pip install pytorch-metric-learning==1.0.0