How to use LinearTruncatedFidelityKernel in Multi-Fidelity optimization

[samuelkim16]

I am doing multi-fidelity optimization using the SingleTaskMultiFidelityGP with default options. Reading through the documentation of LinearTruncatedFidelityKernel, I have a few questions:

• Because of the deterministic part of the kernel, does that mean the actual values of the fidelity parameter matter for degree of correlation? I.e. fidelities of 0 and 1 will be uncorrelated in the kernel, or if I suspect that my simulations are highly correlated, then setting them close to each other e.g. 0.9 and 1 will tell the kernel to incorporate a high degree of correlation.
• If I suspect that my 2 fidelities have a highly linear correlation (i.e. y2=m*y1+b), is there a way to constrain the kernel to be linear?
• Is there a reference where I can read more about this kernel?

[Balandat]

Because of the deterministic part of the kernel, does that mean the actual values of the fidelity parameter matter for degree of correlation? I.e. fidelities of 0 and 1 will be uncorrelated in the kernel, or if I suspect that my simulations are highly correlated, then setting them close to each other e.g. 0.9 and 1 will tell the kernel to incorporate a high degree of correlation.

Yes, that is the inductive bias of the kernel (though it's been a while since I've thought through this in great detail).

If I suspect that my 2 fidelities have a highly linear correlation (i.e. y2=m*y1+b), is there a way to constrain the kernel to be linear?

Not directly, but it shouldn't be hard to implement a modified kernel that encodes this knowledge.

Is there a reference where I can read more about this kernel?

I don't think this exact kernel has a reference, but it is a variant of the multi-fidelity kernels discussed in Section 5.3 of https://arxiv.org/pdf/1903.04703

[ToennisStef]

Hi i also have a question regarding the LinearTruncatedFidelityKernel :

is this kernel really positive definite and a therefore valid kernel for GP's?
if we are only considering the data fidelity the kernel basically boils down to:
$$k_{LT}(x,x')= k_{0}(x_{data},x_{data}') + ((1-x_{fidelity})(1-x_{fidelity}')(1+x_{fidelity}x_{fidelity}'))^p k_{1}(x_{data},x_{data}')$$

with:

• $k_{0}$ and $k_{1}$ being matern kernels
• $x,x'$ being the original training values containing both the fidelity values and the x-function values
• $x_{data},x_{data}'$ being only the x-function values of $x,x'$
• $x_{fidelity},x_{fidelity}'$ being only the fidelity values of $x,x'$

And when evaluating psd'ness of the kernel it is sufficient to look at this this part:
$$((1-x_{fidelity value})(1-x_{fidelity value}')(1+x_{fidelity value}x_{fidelity value}'))^p$$
since the matern kernels are psd
Additionally this part basically boils down to the polynomial kernel:
$$k_{Poly}(x,x')=(1-x'x)^{p}$$
since $x_{fidelity value}$ are clamped to the domain $[0,1]$

Now the polynomial kernel is psd for positive natural numbers. But the power hyperparameter is allowed to be a positive real number. Therefore im asking if the kernel is truely psd?
I would be thrilled if someone can confirm this or this or proof me wrong, or tell me if and why the Linear truncated kernel is a valid kernel. My knowledge in functional analysis is too bad to give proof that the polynomial kernel is not psd for the power hyperparameter not beeing a natural number.

Im asking all this since i am getting a covariance matrix for test data that is not psd and i therefore cannot compute outputs for the SingleTaskMultifidelityGP model

my training data is:

tensor([[0.0416, 0.0000],
        [0.9448, 0.0000],
        [0.6577, 0.0000],
        [0.3248, 0.0000],
        [0.4455, 0.0000],
        [0.5369, 0.0000],
        [0.8162, 0.0000],
        [0.1701, 0.0000],
        [0.2493, 0.0000],
        [0.7682, 0.0000],
        [0.4777, 1.0000],
        [0.8899, 1.0000],
        [0.6402, 1.0000],
        [0.2281, 1.0000],
        [0.1000, 1.0000],
        [0.5005, 1.0000],
        [0.7508, 1.0000],
        [0.3504, 1.0000],
        [0.2531, 1.0000],
        [0.8488, 1.0000],
        [0.6844, 2.0000],
        [0.0742, 2.0000],
        [0.4611, 2.0000],
        [0.7960, 2.0000],
        [0.9415, 2.0000],
        [0.3000, 2.0000],
        [0.1572, 2.0000],
        [0.5858, 2.0000],
        [0.5543, 2.0000],
        [0.1882, 2.0000],
        [0.5264, 3.0000],
        [0.0000, 3.0000],
        [0.1803, 3.0000],
        [0.3819, 3.0000],
        [0.7391, 3.0000],
        [0.7312, 3.0000],
        [0.7375, 3.0000],
        [0.1281, 3.0000],
        [0.7424, 3.0000],
        [0.7449, 3.0000],
        [1.0000, 3.0000]], dtype=torch.float64)

and this code gets me the negative eigenvalues for the covariance matrix of the LinearTruncatedFidelityKernel:

import torch
from botorch.utils.transforms import normalize
from botorch.models.kernels.linear_truncated_fidelity import (
    LinearTruncatedFidelityKernel,
)
from gpytorch.priors.torch_priors import GammaPrior

tdims= [1]
leading_dims= []
dim= 2
nu= 2.5
aug_batch_shape= torch.Size([])

kern = LinearTruncatedFidelityKernel(
                    fidelity_dims=leading_dims + tdims,
                    dimension=dim,
                    nu=nu,
                    batch_shape=aug_batch_shape,
                    power_prior=GammaPrior(3.0, 3.0),
                )

train_X_N = normalize(train_X, bounds=torch.tensor([[0., 0.],[1., 3.]], dtype=torch.float64))
res_kern_N = kern(train_X_N)
real_res_N = res_kern_N.to_dense()
eigenvalues_kern_N = torch.linalg.eigvals(real_res_N)
print("Eigenvalues of the kernel with normalized inputs:", eigenvalues_kern_N)