What performance issues exist when running MCMC on standard GPyTorch models?

[chrisyeh96]

I am interested in implementing a fully-Bayesian GP for Bayesian optimization, similar to the SaasBO implementation in BoTorch. I noticed the following comment in the SaasBO code:

botorch/botorch/models/fully_bayesian.py

Lines 20 to 21 in de46059

	We use a lightweight PyTorch implementation of a Matern-5/2 kernel as there are
	some performance issues with running NUTS on top of standard GPyTorch models.

Could someone please elaborate what sorts of "performance issues" you saw, and why implementing your own Matern-5/2 kernel was able to resolve those issues? I'd like to make sure I don't run into the same issues.

From what I gather, "running NUTS on top of standard GPyTorch models" seems to refer to using GPyTorch's kernel.register_prior() feature, as described in this GPyTorch tutorial:
https://docs.gpytorch.ai/en/stable/examples/01_Exact_GPs/GP_Regression_Fully_Bayesian.html

Tagging @dme65 who implemented SaasBO and @esantorella who wrote the comment in the code.

Also tagging @RaulAstudillo06 and @james-bowden who may be interested in this discussion.

Thank you in advance for your help!

[Balandat]

gpytorch uses linear_operator for structured linear algebra under the hood. This is great (and necessary) in the large data regime to speed up / reduce memory usage of linear algebra operations, but it adds some overhead. That overhead can become significant for small data settings, and adds up if you run this repeatedly as part of an MCMC chain. In particular b/c it's not straightforward to apply JIT compilation to the linear_operator setup which can help a lot in this setting.

I personally haven't revisited this in a while so this information may potentially be outdated (but I think it's still accurate).