Add preconditoner routines to TimeIntegrator with SUNDIALS support

[balos1]

Summary

This PR adds set_preconditioner and set_fast_preconditioner routines to the BaseIntegrator class which allow for preconditioner setup and solve callbacks to be attached and used with SUNDIALS implicit time integrators.

Additional background

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

CC @gardner48

Tutorial (PR soon).

[ajnonaka]

Thanks @balos1 ! I will create a tutorial with a simple custom preconditioner to test this.

[balos1]

Thanks @balos1 ! I will create a tutorial with a simple custom preconditioner to test this.

https://github.com/sundials-codes/amrex-tutorials/tree/advdiff_sundials_hypreprecon/ExampleCodes/SUNDIALS/AdvDiff-MLMGPrecon

[WeiqunZhang]

• Found one correctness issue in PR #5513 at 7823a8b.

• [P2] Changing preconditioner callbacks after stepping causes a segfault — AMReX_SundialsIntegrator.H:248–256
  (

  amrex/Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H

  Lines 248 to 256 in 7823a8b

  	SUNLinSolFree(LS);
  	LS = SUNLinSol_SPGMR(y_data,
  	preconditioning,
  	max_linear_iters, sunctx);
  	AMREX_ALWAYS_ASSERT(LS != nullptr);
  	int flag = 0;
  	if (use_ark) {
  	flag = ARKStepSetLinearSolver(arkode_mem, LS, nullptr);

  ). Calling set_preconditioner() after the first
  solve causes the next evolve() or advance() to replace GMRES without initializing its workspace. GDB confirms the next solve dereferences a null V array. The fast-preconditioner path has the
  same defect. Preserve the existing solver or ensure replacement triggers the required initialization.

Reproduced with CPU SUNDIALS 7.7.0 across DIRK, IMEX-RK, and all three MRI configurations; unchanged callbacks pass. Test harness (/tmp/amrex-pr5513-test/probe.cpp) and GDB trace (/tmp/amrex-
pr5513-test/trace.log).

probe.cpp

#include <AMReX.H>
#include <AMReX_TimeIntegrator.H>
#include <cmath>
using namespace amrex;
int main(int argc, char** argv) {
 Initialize(argc,argv);
 {
 BoxArray ba(Box(IntVect(0),IntVect(3))); DistributionMapping dm(ba);
 MultiFab y(ba,dm,1,0), out(ba,dm,1,0); y.setVal(1.0);
 TimeIntegrator<MultiFab> ti(IntegratorTypes::Sundials,y);
 auto rhs=[](MultiFab& f,MultiFab& s,Real){MultiFab::Copy(f,s,0,0,1,0);f.mult(-1.0);};
 auto zero=[](MultiFab& f,MultiFab&,Real){f.setVal(0.0);};
 ti.set_rhs(rhs); ti.set_imex_rhs(rhs,zero); ti.set_fast_rhs(rhs);
 ti.set_time_step(0.01); ti.set_fast_time_step(0.001);
 int setup=0,solve=0,fs=0,fp=0; int mode=0; ParmParse().query("mode",mode);
 auto ps=[&](MultiFab&,MultiFab&,Real,bool,bool& jcur,Real){++setup;jcur=true;};
 auto pv=[&](MultiFab& z,MultiFab& r,MultiFab&,MultiFab&,Real,Real gamma,Real,int){++solve;MultiFab::Copy(z,r,0,0,1,0);z.mult(1.0/(1.0+gamma));};
 auto fps=[&](MultiFab&,MultiFab&,Real,bool,bool& jcur,Real){++fs;jcur=true;};
 auto fpv=[&](MultiFab& z,MultiFab& r,MultiFab&,MultiFab&,Real,Real gamma,Real,int){++fp;MultiFab::Copy(z,r,0,0,1,0);z.mult(1.0/(1.0+gamma));};
 ti.set_preconditioner(ps,pv); ti.set_fast_preconditioner(fps,fpv);
 if(mode==2) ti.advance(y,out,0.0,0.01); else ti.evolve(out,0.01);
 Print()<<"initial "<<out.min(0)<<" setup="<<setup<<" solve="<<solve<<" fast_setup="<<fs<<" fast_solve="<<fp<<"\n";
 if(mode!=1) {ti.set_preconditioner(nullptr,pv); ti.set_fast_preconditioner(nullptr,fpv);}
 if(mode==2) ti.advance(out,y,0.01,0.01); else ti.evolve(out,0.02);
 Print()<<"replace "<<out.min(0)<<" setup="<<setup<<" solve="<<solve<<" fast_setup="<<fs<<" fast_solve="<<fp<<"\n";
 if(mode!=1) {ti.set_preconditioner(nullptr,nullptr); ti.set_fast_preconditioner(nullptr,nullptr);}
 if(mode==2) ti.advance(y,out,0.02,0.01); else ti.evolve(out,0.03);
 Print()<<"disable "<<out.min(0)<<" setup="<<setup<<" solve="<<solve<<" fast_setup="<<fs<<" fast_solve="<<fp<<"\n";
 }
 Finalize();
}

trace.log

Breakpoint 1 (SUNLinSolInitialize_SPGMR) pending.
Function "SUNLinSolSolve_SPGMR" not defined.
Breakpoint 2 (SUNLinSolSolve_SPGMR) pending.

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
Initializing SUNDIALS with 1 threads...
SUNDIALS initialized.
SPGMR initialize
SPGMR solve V=0x55555587cb10
SPGMR solve V=0x55555587cb10
SPGMR solve V=0x55555587cb10
SPGMR solve V=0x55555587cb10
SPGMR solve V=0x55555587cb10
initial 0.9900498337 setup=1 solve=10 fast_setup=0 fast_solve=0
SPGMR solve V=(nil)

Program received signal SIGSEGV, Segmentation fault.
SUNLinSolSolve_SPGMR (S=0x555555878d10, A=<optimized out>, x=0x55555587e400, b=<optimized out>, delta=0.0020000000000000005) at /home/wqzhang/tmp/sundials-7.7.0/src/sunlinsol/spgmr/sunlinsol_spgmr.c:552
warning: Source file is more recent than executable.
552	  N_VScale(ONE, vtemp, V[0]);
#0  SUNLinSolSolve_SPGMR (S=0x555555878d10, A=<optimized out>, x=0x55555587e400, b=<optimized out>, delta=0.0020000000000000005) at /home/wqzhang/tmp/sundials-7.7.0/src/sunlinsol/spgmr/sunlinsol_spgmr.c:552
#1  0x00007ffff7f51699 in arkLsSolve (ark_mem=0x555555874f10, b=0x5555558763b0, tnow=0.014999999999999999, ynow=0x555555880330, fnow=0x55555587b2c0, eRNrm=<optimized out>, mnewt=0) at /home/wqzhang/tmp/sundials-7.7.0/src/arkode/arkode_ls.c:3451
#2  0x00007ffff7f27965 in arkStep_NlsLSolve (b=0x5555558763b0, arkode_mem=<optimized out>) at /home/wqzhang/tmp/sundials-7.7.0/src/arkode/arkode_arkstep_nls.c:529
#3  0x00007ffff7f89a23 in SUNNonlinSolSolve_Newton (NLS=0x555555876380, y0=<optimized out>, ycor=0x555555875de0, w=0x555555876a40, tol=0.10000000000000001, callLSetup=0, mem=0x555555874f10) at /home/wqzhang/tmp/sundials-7.7.0/src/sunnonlinsol/newton/sunnonlinsol_newton.c:243
#4  0x00007ffff7f29018 in arkStep_Nls (ark_mem=ark_mem@entry=0x555555874f10, nflag=<optimized out>) at /home/wqzhang/tmp/sundials-7.7.0/src/arkode/arkode_arkstep_nls.c:428