Choosing a surrogate model: Any resources for understanding tradeoffs?

[Padarn]

Hey all - thanks for the great work on botorch.

It's awesome to have a library that is agnostic to the surrogate model that is being used to define the underlying probability distribution, but clearly, GP models are the most heavily supported family.

I'm looking for some material to understand the tradeoffs when choosing a surrogate model. I realize the clear answer here is probably "it depends on what your priors are on the function you are trying to model", but to answer that question it would be helpful to know what aspects of prior information I should be thinking about the most.

Appreciate any suggestions for reading or thoughts from the team.

[Balandat]

Hi, and thanks for your interest.

And yes, the answer is, as usual, a resounding "it depends". Not only on your priors, but also (related to the priors) the properties of the function, things like the amount of data you have, what your requirements are in terms of how quickly you'd like to fit the model and make predictions etc. Are you looking at a particular kind of problem space or are you just generally curious?

This is a very open question indeed, but I guess at the high level there are two generally valid points:

1. Your model needs to be a probabilistic one (it doesn't necessarily need to be able to characterize the distribution explicitly so long as you can sample from it)
2. The more domain knowledge you can encode the more sample efficient you can be. This could either be on the (hyper-)priors of something like a GP (e.g. reasonable beliefs about length scales) or about the structure of the function itself (e.g. maybe you have some domain knowledge telling you that the behavior you're trying to model is linearity in some of the parameters).

I hope this helps.

[Padarn]

Hey @Balandat thanks for taking the time to respond. I'll follow up on a few things you mention below, but let me know if the GitHub discussions are not the right place for this.

Are you looking at a particular kind of problem space or are you just generally curious?

I came to the library because we're working on problems in adaptive experiments (A/B testing). In particular, I was trying to reason through how we build different sources of uncertainty into our model (we expect both variances caused by sampling, underlying environment changes, and unobserved factors).

But my goal was to build some intuition so I can reason about that myself, rather than ask for specific practical advice.

the properties of the function

Yep, this makes sense. My basic understanding is that (with the common kernels) Gaussian Processes are most effective when you have very little prior information about the function properties but expect a lot of smoothness. Is there something else about them that I'm missing which seems to make them very popular in the bayesian optimisation problem?

what your requirements are in terms of how quickly you'd like to fit the model and make predictions etc

This is a good point. I had thought about the scalability of Gaussian Processes vs models like Bayesian NN, but only in terms of the training speed - Having now experimented a bit I also see the prediction step is an equally important property to consider.

This could either be on the (hyper-)priors of something like a GP (e.g. reasonable beliefs about length scales)

Not a factor I had tried to think about yet. Thanks for the example.

I hope this helps.

It does. Thanks!

[Balandat]

My basic understanding is that (with the common kernels) Gaussian Processes are most effective when you have very little prior information about the function properties but expect a lot of smoothness. Is there something else about them that I'm missing which seems to make them very popular in the bayesian optimisation problem?

One other thing is that they usually yield well-calibrated uncertainty estimates. The ones you'll get from other models such as BNNs can be a bit iffy and hard to get right.

[Padarn]

That matches my experience too, thanks for the reminder.