Rules for LinearSolve.jl

[yebai]

This is required by models like: https://turinglang.org/docs/tutorials/10-bayesian-differential-equations/, or maximum_likelihood based learning.

Probably need to

• add Mooncake support to: https://github.com/SciML/SciMLSensitivity.jl
  • Testing use of rrule SciML/DiffEqBase.jl#1103
  • Add Mooncake to Alternative AD Frontends SciML/SciMLSensitivity.jl#1151
  • Mooncake Inside Problems SciML/SciMLSensitivity.jl#1152
  • Setup Mooncake rrule SciML/DiffEqBase.jl#1158
  • Get Mooncake direct adjoints working SciML/OrdinaryDiffEq.jl#2723
  • customised error message: @mooncake_overlay does not seem to activate if in an extension #587 + Mooncake overlay for erroring out before Mooncake.DerivedRule construction SciML/DiffEqBase.jl#1169
• add support for Optimization.jl (see, here and here): now works, see here.
• Adding Mooncake to the AD list.  SciML/Optimization.jl#936
• add more rules requested here
  • NNLib
  • NonlinearSolve
  • LinearSolve
  • ARPACK

Related:

• Adding A New AD SciML/SciMLSensitivity.jl#1105
• Support more AD backends SciML/BoundaryValueDiffEq.jl#288

[ChrisRackauckas]

The DiffEq integration is done. Setting up overloads on NonlinearSolve and LinearSolve would help a lot as well, but at least ODEs should be fine. There are some edge cases with callbacks but that's due to the type expectations which maybe can be relaxed?

[yebai]

@ChrisRackauckas, can you clarify the role of NonlinearSolve and LinearSolve for SciML? Also, why are NNLib and ARPACK required?

[ChrisRackauckas]

They are the linear solvers and nonlinear solvers. NNLib is required for any machine learning usage as that's what Flux and Lux use for kernels. ARPACK is an eigenvalue solver for sprase eigenvalues, we can probably skip that one if we go through the LinearSolve.jl interface for all times.