MINLP Optimization

[marcosfelt]

I saw #772 which implements mixed domain GPs (categorical + continuous). Is there any interest in integrating a MINLP optimizer in the botorch.optim module to enable cases with lots of categorical dimensions?

Currently, we implemented a simple wrapper around the standard acquisition function that rounds every input to a one-hot encoding (see here). However, with this new functionality, it would be possible to consider using a MINLP solver directly.

[Balandat]

Yes, that would definitely be of interest. Do you have a specific solver for this in mind?

In the past we've found random initialization + optimization of continuous parameters + nearest neighbor search to work decently well in many cases. But of course doing it "properly" would be good.

One of the challenges i can foresee is that the GP models are optimized for batch evaluation, whereas most off-the-shelf optimizers are not and so might be quite slow b/c of the slow function evaluation of the acquisition function at individual points.

[marcosfelt]

@Balandat A super slow response on my end! Are you doing nearest neighbor search for categorical variables? I am assuming that means you have some sort of embedding?

[Balandat]

So the most basic local search for categoricals we (and @dme65 more specifically) have tried in the past was based on just mutating single parameters (i.e. sth like NN in the Hamming distance).

But we (in particular @sdaulton) have also played around with embeddings for the categoricals, so a NN search based on that is also possible (and more promising if the embedding is decent). But that of course requires actually using the embedding in the model; afaik we haven't open sourced any of this yet.