Remove AD backend loops in test suite

[penelopeysm]

Closes #2307

Summary

In Turing's test suite, we have historically run HMC, etc. tests three times, once with ForwardDiff, once with ReverseDiff, and once with Mooncake. This PR removes this loop and only runs sampler-based tests with ForwardDiff.

In its place, I've moved all the AD-based tests to test/ad.jl. These comprise:

ADTypeCheckContext tests

These are directly copied from the old test/test_utils/ad_utils.jl file, and basically exist to make sure that samplers and optimisers don't suddenly default to using ForwardDiff.

SamplingContext tests with DynamicPPL demo models

When running sample(model, spl, N), the LogDensityFunction gets constructed with a context of SamplingContext(rng, spl), which means that model execution can in general go down a completely different path when compared to just using DefaultContext. Thus, there is still a need to test the correctness of AD on this.

GibbsContext tests with DynamicPPL demo models

The same rationale as above, but for Gibbs sampling, which contextualises the model with GibbsContext.

Are the demo models enough?

Given that we would previously have been testing AD on a range of random models in the inference tests, it might be fair to ask the question of whether we are losing AD testing coverage by not testing on all of the models.

I've (rather painstakingly) gone through all of the models in the inference tests, and it turns out that most of them are not very interesting. For example, gdemo_default is used a lot, and that's already one of the DynamicPPL demo models.

Other models are speciically chosen to highlight issues with DynamicPPL internals rather than AD, so I don't really consider losing the AD testing on these to be a huge problem.

Most of the non-trivial Gibbs models have discrete parameters and so obviously aren't suitable for general AD testing.

There were three models where (in my best judgment) were worth adding some tests for, so I've already transferred these to ADTests: TuringLang/ADTests@4c34c19 (I am aware that ADTests is not really a 'test' in the truest sense, as it doesn't get run in CI.)

More generally @yebai has made the point that for AD testing in CI, we can stick to just the DynamicPPL demo models. Although I do still have slight concerns that that they don't offer complete coverage, I do think that they do cover a pretty wide range of DynamicPPL features, so am happy to sign off on this PR, in the interests of not letting the perfect be the enemy of the good.

What next?

This PR roughly halves the time taken to run Gibbs tests, which are the slowest tests.

I think it would be a very quick win to try to rebalance the test suite in CI, especially if we can split the Gibbs tests up. Right now the AD tests are being run together with 'everything else', which is not optimal.

I also note that the Gibbs demo model tests take an hour to run on 1.10, but around 15 minutes on 1.11. This is a huge source of the 90-minute runtime for the slowest tests (Gibbs on 1.10). I don't know why this is the case. In an ideal world, I'd like somebody to look into it, although we don't have much time.

[github-actions]

Turing.jl documentation for PR #2564 is available at:
https://TuringLang.github.io/Turing.jl/previews/PR2564/

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.86%. Comparing base (dea5d19) to head (f1263bf).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2564   +/-   ##
=======================================
  Coverage   85.86%   85.86%           
=======================================
  Files          21       21           
  Lines        1429     1429           
=======================================
  Hits         1227     1227           
  Misses        202      202           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coveralls]

Pull Request Test Coverage Report for Build 15325551548

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 85.924%

---

Totals
Change from base Build 15324242745:	0.0%
Covered Lines:	1227
Relevant Lines:	1428

---

💛 - Coveralls

[mhauru]

I, too, am a bit nervous about how much this reduces our test coverage (and I don't mean lines of code but functionality), but it definitely seems like the right way to do things long term.

One concern I have is if some bits of code that aren't within logdensity_and_gradient calls might still somehow fail, due to e.g. not handling tracer types. For instance, Gibbs does a bunch of resetting of sampler states, and nasty things could happen if some vector element types get messed up. run_ad wouldn't catch that, since it evaluates the LDF without calling step. Maybe in the loop over algorithms in ad.jl we could also do a call to sample with each AD backend and model, just to ensure that it doesn't crash?

[penelopeysm]

I also removed this (outdated) check for dot assume so now it checks with every sampler combination

[sunxd3]

I am not very concerned with the potential coverage lost - we can add tests back when things fail.
The changes makes a lot of sense for me. All look good.

(Is this missed?)

[sunxd3]

Also, we can probably get rid of the adbackend and just use Turing.DEFAULT_ADTYPE? Not essential.

Turing.jl/test/mcmc/gibbs.jl

Line 376 in 5e7ed7d

adbackend = Turing.DEFAULT_ADTYPE

Turing.jl/test/mcmc/hmc.jl

Line 22 in 5e7ed7d

adbackend = Turing.DEFAULT_ADTYPE

Turing.jl/test/mcmc/Inference.jl

Line 23 in 5e7ed7d

adbackend = Turing.DEFAULT_ADTYPE

[penelopeysm]

(Is this missed?)

To be completely honest, I have no idea what that test is doing or why it's using a different adtype.

I didn't want to touch it in this PR, but let's put it this way, if someone else were to put in a PR removing this test, I would probably approve...

Also, we can probably get rid of the adbackend and just use Turing.DEFAULT_ADTYPE?

We could, or we could just drop the adtype keyword argument entirely and let it default to the default (i.e. just use NUTS() rather than NUTS(; adtype=Turing.DEFAULT_ADTYPE). I guess it's a question of whether to be explicit or not -- in this case I erred on being explicit. But honestly, it'd be a lot cleaner to not have it in there, so I'll remove them and then merge.