Some notes on the comment in kernel.py

[Zhaoyi-Yan]

https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/kernel.py#L28
Although in this script, it says that when
pytorch/pytorch#25799
is merged, the Distance class can be replaced with torch.cdist.
However, even though the pull request is merged, current torch.cdist and torch.pdist still suffer from bugs on hanlding large batchsize.
See
pytorch/pytorch#31593
pytorch/pytorch#31167

So I try this funciton, and it works fine!
Thank you, guys!

[jacobrgardner]

Hi @Zhaoyi-Yan -- Yeah, at the moment none of our code relies on torch.cdist precisely because it has been a little unstable for a couple of releases now. That comment is a little old at this point, but i think the general point is a reminder to ourselves to really make sure that torch.cdist is working before switching to it again.

I think our code has gone through 2-3 iterations of using torch.cdist, and each time we disabled it when we realized there was some edge case that is important to us but is either not performant or a little buggy with the torch version.

The other issue with the built in version is that our only option to get squared distances is by computing the distance and then squaring it, which is actually pretty unstable because of the square root that already happened to get the actual distance. So, until torch supports a "squared distance" option to torch.cdist, we probably won't be able to switch to it without getting nans in our backward passes from time to time.

[Balandat]

I requested the squared (p-ed) distances in pytorch/pytorch#28119, doesn't seem like much has happened on this :(