Series of tests for AD

[theogf]

I started to re-adapt the tests given the new structure and make them look a bit nicer.
Right now the idea is to compare the results (when possible) with FiniteDifferences.jl
The tests are quite generic and changeable.
The idea is to test differentiability given the inputs for all ADs and to test the differentiability given the kernel parameters only given Zygote. (other kernels would require more complex solutions)
The tests functions for AD are contained in test/utils_AD.jl.
A first function needs to be called with FiniteDifferences.jl to check that it runs without errors (eliminate one source of error when comparing)
A second function will compare a AD (so far implemented for Zygote, ForwardDiff and ReverseDiff) with a kernelfunction which may take arguments. The test setup for a kernel would look like this:

kfunction = (l -> transform(SEKernel(), first(l)))
args = [2.0]
v = test_FiniteDiff(kfunction, args)
if !v.anynonpass
    for AD in [:Zygote, :ForwardDiff, :ReverseDiff]
        test_AD(AD, kfunction, args)
    end
end

where k = kernelfunction(args) and args is a vector
Alternatively for no args :

kfunction = SqExponentialKernel
v = test_FiniteDiff(kfunction)
if !v.anynonpass
    for AD in [:Zygote, :ForwardDiff, :ReverseDiff]
        test_AD(AD, kfunction)
    end
end

Are tested :

• kappa for SimpleKernel
• k(x, y) given k, x AND y
• kernelmatrix(A, B) for both RowVecs and ColVecs, also given A, B and k (I use sum to get a scalar on it)
  kernelmatrix(A)` same as above

Are added to make more tests pass:

• Indirection to _map from Base.map for avoid Zygote bug with map on AbstractVector
• More adjoints for DotProduct and Sinus
• Fixes for some function to be able to differentiate through (especially Matern Kernel)

[theogf]

One major issue is currently #113. And I have no idea how to solve it

[willtebbutt]

To be honest, I was planning to address this at the same time as #88.

In my opinion the correct way to do this is to incorporate differentiability tests in with the standard test suite, as opposed to having a collection of tests in a separate location.

No objection to this going in for now to get us in roughly the right place, but I don't think it's the correct solution long-term.

[theogf]

So you propose to integrate each all the AD test separately for each type of kernel in their test file?
It sounds good to me, but what I show for now is what do we even want to test. This is a very rough draft, also to explore what is currently failing.

[willtebbutt]

So you propose to integrate each all the AD test separately for each type of kernel in their test file?

Yup.

but what I show for now is what do we even want to test.

We want to test everything in the public-facing API for every kernel that we write tests for.

This is a very rough draft, also to explore what is currently failing.

👍

[theogf]

@willtebbutt Would you be satisfied by these tests? I would lay them out all over the tests but I prefer to check with you first :P

[willtebbutt]

It's definitely a reasonable starting point. We shouldn't really be testing gradients, rather we should be testing pushforwards / pullbacks. Unfortunately ForwardDiff and ReverseDiff don't expose these in their public APIs, so I'll need to do some digging after NeurIPS.

I would propose you merge this, because it's a definitely improvement over what we currently have, and we revisit after NeurIPS when refactoring the test suite.

edit: I'm not sure why coverage has reduced, but it would be good to understand why before merging.

[theogf]

Coverage has reduced because of some additional adjoints and functions that were needed for AD. Once I put all the tests in place it should actually go up :)

[theogf]

So I spread all the tests to the BaseKernels, Kernels and Transforms and when tests were failing I simply left out the AD in question and left a @test_broken with a comment. I listed all the failures in #116

[theogf]

I was also unable to find the proper adjoint for the pairwise function with the Sinus metric

[devmotion]

Tests are failing but not because of new tests (some issue with the FBM kernel).

I'm surprised about these test failures - it has to be related to the PR in some way, otherwise they wouldn't have passed all the time on master? Maybe try to set a seed (if we haven't fixed one yet)?

[willtebbutt]

I'm surprised about these test failures - it has to be related to the PR in some way, otherwise they wouldn't have passed all the time on master? Maybe try to set a seed (if we haven't fixed one yet)?

Yeah, would definitely be nice to resolve this before merging.

[theogf]

I will try to figure out the issue with FBM, but I thought this was maybe irrelevant to this PR ?

[theogf]

Can I merge this?

[theogf]

Ah maybe before I can put all dependencies for the test directly in a Project.toml

[theogf]

All tests are passing, if there is no objections I am going to merge

[willtebbutt]

@theogf do you know why coverage has dropped?

[theogf]

@willtebbutt, yes, I needed to add some functionalities (mostly for Zygote) but sometimes it was not enough to make all tests pass (yet I left them there as they will be needed in the future). Right now tests are either all correct and removed. I want to tackle all of the different AD problems (all mentioned in #116 ) but would prefer to go one PR at a time, ss there are definitely a lot of issues.