alpha-ctsm5.4.CMIP7.19.ctsm5.3.082: Fix LII2 COMPARE_base_no_interp failures

[slevis-lmwg]

Description of changes

LII2 tests require some steps in order to work when we update fsurdat files and/or when we update finidat files (details in this post). This PR handles those steps.

Unlike in previous updates (e.g. ctsm5.2 and ctsm5.3), when I updated for ctsm5.4, COMPARE_base_no_interp continued to fail in the LII2 tests due to two diagnostic fields appearing unnecessarily in restart files (details in #3252). This PR removes these two fields from restarts.

• ./rimport clmi.f09_interp_from.ctsm5.4.CMIP7_ciso_ctsm5.3.075_f09_124_pSASU.clm2.r.0161_c251118.nc
• relink.py

Specific notes

CTSM Issues Fixed (include github issue #):
Fixes #3252

Are answers expected to change (and if so in what way)?
Yes, because the LII2 tests point to new finidat files.

Any User Interface Changes (namelist or namelist defaults changes)?
Yes, to enable LII2 tests to pick up the correct finidat files.

Testing performed, if any:
The following now passes:
./create_test LII2FINIDATAREAS_D_P256x2_Ld1.f09_t232.I1850Clm60BgcCrop.derecho_intel.clm-default -c /glade/campaign/cgd/tss/ctsm_baselines/alpha-ctsm5.4.CMIP7.17.ctsm5.3.082
The other LII2 test looks the same with the suffix --clm-matrixcnOn_ignore_warnings added to it, so I expect it to pass.

[slevis-lmwg]

derecho and izumi
FAIL ./run_sys_tests -s aux_clm -c alpha-ctsm5.4.CMIP7.17.ctsm5.3.082 -g alpha-ctsm5.4.CMIP7.19.ctsm5.3.082
derecho only
IN PROG ./run_sys_tests -s ctsm_sci -c ctsm_sci-alpha-ctsm5.4.CMIP7.17.ctsm5.3.082 -g ctsm_sci-alpha-ctsm5.4.CMIP7.19.ctsm5.3.082

So far I have only looked at aux_clm on derecho:

1. OK NLCOMP diffs that lead to BASELINE diffs in use_init_interp = .true. cases (mostly f10 and 1x1, but also an f09) where finidat has changed as intended:

BASE: finidat = ctsm5.4.CMIP7_ciso_ctsm5.3.075_f09_124_pSASU.clm2.r.0161-01-01-00000.nc'
COMP: finidat = clmi.f09_interp_from.ctsm5.4.CMIP7_ciso_ctsm5.3.075_f09_124_pSASU.clm2.r.0161_c251118.nc'

2. OK BASELINE diffs in cases with use_init_interp = .false. where only VEGWP[LN,PD] have changed in their fill patterns. The change is expected according to this post because I removed these two diagnostic fields from restart.
3. OK COMPARE_base expected failures in an ERI, a PEM, and a REP test.
4. FAIL COMPARE_base diffs where only the two fields VEGWP[LN,PD] have changed; this DOES NOT go away when rerunning and comparing against this PR's baseline (alpha-ctsm5.4.CMIP7.19.ctsm5.3.082), suggesting that these two fields need to be on restart for some reason...
5. FAIL RUN because "Cannot interpolate from a run without c13/c14 to a run with c13/c14"

FAIL ERI_D_Ld9.f10_f10_mg37.I1850Clm60BgcCrujra.derecho_gnu.clm-default RUN
FAIL SETPARAMFILE_Ld5.f10_f10_mg37.I1850Clm60BgcCrujra.derecho_gnu.clm-default RUN
FAIL SMS_Lm1.f10_f10_mg37.I1850Clm60BgcCropCmip6waccm.derecho_gnu.clm-basic RUN
FAIL ERS_Ly3.f10_f10_mg37.I1850Clm60BgcCropCmip6.derecho_intel.clm-basic RUN
FAIL SMS_Ld2_D_PS.f09_g17.I1850Clm60BgcCropCmip6.derecho_intel.clm-basic_interp RUN

Get these to point to the other f09 file OR (easier?) rerun LII2 as a Bgc test with c13/c14.

[slevis-lmwg]

@olyson turns out that you were correct that we still have a problem with the two fields VEGWP[LN,PD], though I only confirmed that when I reran a test against the newly generated baseline (see 4 above) and still had DIFFs in these fields. This means that these fields do need to go on restart files. This is my plan:

1. I will look at the code in case I find why these depend on restart, because I would not have expected that. If that doesn't lead me anywhere,
2. I could try bisecting as I said yesterday to investigate when this became a problem. Though...
3. I could just pull these two fields from history and open an issue explaining the situation, and this would be simplest.

There is another set of failures (see 5 above) that I need to address. I have a clearer plan for that, so I may start with that. (After I finish "export compliance" training...)

[olyson]

Thanks @slevis-lmwg . As we discussed, it isn't clear why none of the local noon solar variables have to be on the restart file. In looking at the code further though, it might be because they are set to spval at every non-local noon time step in SurfaceRadiationMod.F90, whereas vegwp_ln seems to be only set to spval at every non-local noon time step if it is daytime. Maybe it could be set at the end of PhotosynthesisHydraulicStress so that it is set at every time step. As you suggest, maybe the two fields could be removed for now and possible solutions could be experimented with in a new issue.

[slevis-lmwg]

Oh yeah, I still have problem (5) from the list above, so I need to deal with that:

Rebuilt and reran the LII2 test, but with use_c13 and use_c14 = .true., so as to generate a finidat that I hope will work with the failing tests:
/glade/derecho/scratch/slevis/LII2FINIDATAREAS_D_P256x2_Ld1.f09_t232.I1850Clm60BgcCrop.derecho_intel.clm-default.C.20251119_112837_zehcbv/run/init_generated_files

On derecho and izumi
OK ./run_sys_tests -s aux_clm -c alpha-ctsm5.4.CMIP7.17.ctsm5.3.082 -g alpha-ctsm5.4.CMIP7.19.ctsm5.3.082
On derecho only
IN PROG ./run_sys_tests -s ctsm_sci -c ctsm_sci-alpha-ctsm5.4.CMIP7.17.ctsm5.3.082 -g ctsm_sci-alpha-ctsm5.4.CMIP7.19.ctsm5.3.082

ctsm_sci waiting for two RX* tests to finish, else OK

[ekluzek]

This is a new file that was created by interpolating from the original right? It should then have

use_init_interp either removed or set to .false. Same with the one just above.

I think it's also true of this one:

lnd/clm2/initdata_esmf/ctsm5.4/clmi.f19_twiceinterp_from.I1850Clm50BgcCrop-ciso.1366-01-01.0.9x1.25_gx1v7_simyr1850_c251030.nc

[slevis-lmwg]

Ok, I will try that.

[slevis-lmwg]

Removed use_init_interp = .true. from the two interp files (clmi.f09_interp... and clmi.f19_twiceinterp...) and submitted these tests:

./create_test LII2FINIDATAREAS_D_P256x2_Ld1.f09_t232.I1850Clm60BgcCrop.derecho_intel.clm-default -c /glade/campaign/cgd/tss/ctsm_baselines/alpha-ctsm5.4.CMIP7.19.ctsm5.3.082
./create_test LII_D_Ld3_PS.f19_g17.I2000Clm50BgcCrop.derecho_intel.clm-default -c /glade/campaign/cgd/tss/ctsm_baselines/alpha-ctsm5.4.CMIP7.19.ctsm5.3.082

[slevis-lmwg]

Tests PASS! Submitting the test-suites again now.

izumi
OK aux_clm

derecho
OK ./build-namelist_test.pl
PASS make black and lint
PASS python -u and -s (after updating a file path and file name to the ctsm5.4 values)
IN PROG aux_clm
IN PROG ctsm_sci