model fitting errors for SingleTaskMultiFidelityGP & MultiTaskGP

[ToennisStef]

Hi
During optimization runs using the SingleTaskMultiFidelityGP and the MultiTaskGP i often encounter model fitting errors using the fit_gpytorch_mll function.

I was wondering where these model fitting problems came from. I was also wondering if both models require a strict ordering of the fidelity values? and how do these models handle non-linear inter-task/inter-fidelity dependencies?
For example, I got this test problem with 4 fidelity levels:

Here the SingleTaskMultiFidelityGP had some model fitting issues.
I know that these issues can also arise when there are identical or many close data points, which then leads to an ill-conditioned covariance matrix. But could some problems also arise from the fact that there are non-linear inter-fidelity/inter-task dependencies and/or that the model requires a strict ordering of the fidelity functions, which is not given (meaning target fidelity has lowest function values on the whole domain, second highest fidelity has 2nd lowest function values on the whole domain, etc.)?

To be honest I also set the noise covariance of the models close to zero, which I know can also be troublesome, but I cannot pinpoint the exact problem since the error is not very informative.

Best regards,
Stefan Tönnis

[sdaulton]

SingleTaskMultiFidelityGP does assume that the fidelity of the function is monotonic in the parameter (based on the kernel from https://arxiv.org/abs/1903.04703). You could use a SingletaskGP if the relationship between the fidelity parameter and fidelity of the function is not monotonic. That maybe the source of the model fitting errors. What are the model fitting errors?

[ToennisStef]

A quick followup on this one:

The generall error message when using the fit_gpytorch_mll on the SingleTaskMultiFidelityGP was very uninformative since it was just:
ModelFittingError: All attempts to fit the model have failed.

I wanted to get a general overview of the mll space and therefore plotted the mll for different values of the different hyperparameters of the SingleTaskMultiFidelityGP. However, when setting the hyperparameter values manually I noticed, that i always got a

NotPSDError: Matrix not positive definite after repeatedly adding jitter up to 1.0e-03.

Error message when setting the "power" hyperparameter from the LinearTruncatedFidelityKernel to values below 1. I noticed that the constraint on this hyperparameter is generally a positivity constraint, not a >=1 constraint.
My understanding of the model or rather the LinearTruncatedFidelityKernel is not wellenough to make proper sense of this. I can't tell if its generally problematic if the power hyperparameter of the LinearTruncatedFidelityKernel takes values between 0 and 1, or if its generally possible that this hyperparameter takes values between 0 and 1 and my dataset is causing weird problems.

Also when logging the parameter values during optimization with a callback function. i noticed that the final power hyperparameter value before the ModelFittingError is raised is also below 1. I would therefore assume that this is causing the model fitting error.

Any insight on this is highly appreciated!
Best regards

[ToennisStef]

Now to update again:
I see that the LinearTruncatedFidelityKernel should generally be PSD for power hyperparameter values greater than 0. But when plotting the eigenvalues of the covariance matrix over different power hyperparameter values, i could notice discontinuities:

The orange gradient plot is the gradient of the specific eigenvalue with respect to the raw_power parameter of the model.

My question would be why do these discontinuities occur? Can this be prevented?
I am updating this post as soon as possible with proper code for reproducibility.

[Balandat]

Hmm that is rather interesting indeed. This does look quite clean so it doesn't obviously seem be caused by any numerical issues. Please do share a repro for how you generated the plot.

[ToennisStef]

Hi I have revisited the problem and have some reproducable data. It is at the end of the message since i posted this in another thread (#2706 (comment)):

Hi i also have a question regarding the LinearTruncatedFidelityKernel :

is this kernel really positive definite and a therefore valid kernel for GP's?
if we are only considering the data fidelity the kernel basically boils down to:
$$k_{LT}(x,x')= k_{0}(x_{data},x_{data}') + ((1-x_{fidelity})(1-x_{fidelity}')(1+x_{fidelity}x_{fidelity}'))^p k_{1}(x_{data},x_{data}')$$

with:

• $k_{0}$ and $k_{1}$ being matern kernels
• $x,x'$ being the original training values containing both the fidelity values and the x-function values
• $x_{data},x_{data}'$ being only the x-function values of $x,x'$
• $x_{fidelity},x_{fidelity}'$ being only the fidelity values of $x,x'$

And when evaluating psd'ness of the kernel it is sufficient to look at this this part:
$$((1-x_{fidelity value})(1-x_{fidelity value}')(1+x_{fidelity value}x_{fidelity value}'))^p$$
since the matern kernels are psd
Additionally this part basically boils down to the polynomial kernel:
$$k_{Poly}(x,x')=(1-x'x)^{p}$$
since $x_{fidelity value}$ are clamped to the domain $[0,1]$

Now the polynomial kernel is psd for positive natural numbers. But the power hyperparameter is allowed to be a positive real number. Therefore im asking if the kernel is truely psd?
I would be thrilled if someone can confirm this or this or proof me wrong, or tell me if and why the Linear truncated kernel is a valid kernel. My knowledge in functional analysis is too bad to give proof that the polynomial kernel is not psd for the power hyperparameter not beeing a natural number.

Im asking all this since i am getting a covariance matrix for test data that is not psd and i therefore cannot compute outputs for the SingleTaskMultifidelityGP model

my training data is:

tensor([[0.0416, 0.0000],
        [0.9448, 0.0000],
        [0.6577, 0.0000],
        [0.3248, 0.0000],
        [0.4455, 0.0000],
        [0.5369, 0.0000],
        [0.8162, 0.0000],
        [0.1701, 0.0000],
        [0.2493, 0.0000],
        [0.7682, 0.0000],
        [0.4777, 1.0000],
        [0.8899, 1.0000],
        [0.6402, 1.0000],
        [0.2281, 1.0000],
        [0.1000, 1.0000],
        [0.5005, 1.0000],
        [0.7508, 1.0000],
        [0.3504, 1.0000],
        [0.2531, 1.0000],
        [0.8488, 1.0000],
        [0.6844, 2.0000],
        [0.0742, 2.0000],
        [0.4611, 2.0000],
        [0.7960, 2.0000],
        [0.9415, 2.0000],
        [0.3000, 2.0000],
        [0.1572, 2.0000],
        [0.5858, 2.0000],
        [0.5543, 2.0000],
        [0.1882, 2.0000],
        [0.5264, 3.0000],
        [0.0000, 3.0000],
        [0.1803, 3.0000],
        [0.3819, 3.0000],
        [0.7391, 3.0000],
        [0.7312, 3.0000],
        [0.7375, 3.0000],
        [0.1281, 3.0000],
        [0.7424, 3.0000],
        [0.7449, 3.0000],
        [1.0000, 3.0000]], dtype=torch.float64)

and this code gets me the negative eigenvalues for the covariance matrix of the LinearTruncatedFidelityKernel:

import torch
from botorch.utils.transforms import normalize
from botorch.models.kernels.linear_truncated_fidelity import (
    LinearTruncatedFidelityKernel,
)
from gpytorch.priors.torch_priors import GammaPrior

tdims= [1]
leading_dims= []
dim= 2
nu= 2.5
aug_batch_shape= torch.Size([])

kern = LinearTruncatedFidelityKernel(
                    fidelity_dims=leading_dims + tdims,
                    dimension=dim,
                    nu=nu,
                    batch_shape=aug_batch_shape,
                    power_prior=GammaPrior(3.0, 3.0),
                )

train_X_N = normalize(train_X, bounds=torch.tensor([[0., 0.],[1., 3.]], dtype=torch.float64))
res_kern_N = kern(train_X_N)
real_res_N = res_kern_N.to_dense()
eigenvalues_kern_N = torch.linalg.eigvals(real_res_N)
print("Eigenvalues of the kernel with normalized inputs:", eigenvalues_kern_N)