externals update broke NEON features

[wwieder]

Brief summary of bug

previously we were able to run postad cases using run_neon, but that functionality seems to have broken with externals update

General bug information

CTSM version you are using:
Feature works as intended with dev171, but is broken with dev172

Does this bug cause significantly incorrect results in the model's science? [Yes / No]
Yes, removes a helpful functionality in running neon cases

Configurations affected: [Fill this in if known.]
neon cases, but also potentially testing issues (@slevis-lmwg is noting with the sasu PR)

Details of bug

the same command using dev171 works

[~/CTSM/tools/site_and_regional] (dev172)
$./run_neon --neon-sites TOOL --run-type postad --output-root /glade/derecho/scratch/wwieder/neon_AK
---- building a base case -------
---- creating a base case -------
---- base case created ------
---- base case setup ------
---- base case build ------
--- This may take a while and you may see WARNING messages ---
WARNING: No .input_data_list files found in dir 'Buildconf'
^[[O^[[ITime required to building the base case: 427.0643048286438 s.
using this version: latest
---- cloning the base case in /glade/derecho/scratch/wwieder/neon_AK/TOOL.postad
Traceback (most recent call last):
  File "./run_neon", line 48, in <module>
    main(__doc__)
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/run_neon.py", line 238, in main
    experiment,
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/neon_site.py", line 299, in run_case
    self.set_ref_case(case)
  File "/glade/u/home/wwieder/CTSM/tools/site_and_regional/../../python/ctsm/site_and_regional/neon_site.py", line 356, in set_ref_case
    symlink_force(reffile, os.path.join(rundir, os.path.basename(reffile)))
  File "/glade/u/home/wwieder/CTSM/cime/CIME/utils.py", line 1473, in symlink_force
    raise e
  File "/glade/u/home/wwieder/CTSM/cime/CIME/utils.py", line 1467, in symlink_force
    os.symlink(target, link_name)
FileNotFoundError: [Errno 2] No such file or directory: '/glade/derecho/scratch/wwieder/neon_AK/TOOL.ad/run/TOOL.ad.clm2.r.1018-01-01-00000.nc' -> '/glade/derecho/scratch/wwieder/neon_AK/TOOL.postad/run/TOOL.ad.clm2.r.1018-01-01-00000.nc'

Important output or errors that show the problem

Previously noted in #2433, but creating a new issue here, as it's unrelated to the issue @TeaganKing addressed in #2435.

[slevis-lmwg]

SASU PR #640 has a fix for the missing /run directory, but now gets a missing /timing directory.

[slevis-lmwg]

A big picture question:
Do CIME developers test whether their mods break things elsewhere? If not, does it seem that they should?

[ekluzek]

This is a good question @slevis-lmwg. There is internal CIME testing, and if there is functionality that we can isolate and need to be tested, we could add it to that level. However, I don't think that applies here. But, whenever it does it is something that we should make sure we have in place. Although I think it's likely that there should be more testing of CIME internals than is currently being done.

The thing we are doing here is using CIME internals in ways that aren't tested. The testing I mention above could help with that. Or not using things at this internal level. But, I think there's good reasons we have for using things at this level for this work.

The other kind of testing that can help find this in the CESM development process would be adding CESM prealpha testing. This would be caught when new CESM alpha tags are made due to external updates. The thing that we need there is to have a test that runs neon from aux_clm. I thought we had an issue for doing that, but I can't find it.

The thing we have for now is #2276, so I'm make a new issue for this.

[ekluzek]

@TeaganKing it looks like the kind of checking that should be added here (in latest CTSM it will go in tower_site.py rather than run_neon) is to check if reffile exists (or at least it's parent directory), and also that rundir exists and can be written to. So a little bit of extra checking before the symlink_force. It's possible that reffile doesn't exist at the point this is done, so then you just need to make sure it's parent directory exists. This is applying what @slevis-lmwg did to run_neon....

[TeaganKing]

Thanks for pointing this out, @ekluzek . I'll plan to make these changes on the next PLUMBER PR.

[slevis-lmwg]

My most recent test suggests that the code change that I shared earlier today did not fix anything, so it was a false positive. I will continue looking.

[slevis-lmwg]

The latest version of my failing test
SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.20240325_185731_ajymv0
has now passed! I pushed a new commit with the code update to #640. The last two commits combined represent the fix:

--- a/cime_config/SystemTests/sspmatrixcn.py
+++ b/cime_config/SystemTests/sspmatrixcn.py
@@ -207,6 +207,8 @@ class SSPMATRIXCN(SystemTestsCommon):
                      linkfile = os.path.join(rundir, os.path.basename(item))
                      if os.path.exists(linkfile):
                          os.remove( linkfile )
+                     if not os.path.isdir(rundir):
+                         os.makedirs(rundir)
                      os.symlink(item, linkfile )

@ekluzek this had been your original suggestion, i.e. to just make the run directory before calling symlink.

[wwieder]

here's my issue running a transient case with dev175 and dev171

File /glade/derecho/scratch/wwieder/neon_AK/MLBS.transient/LockedFiles/env_build.xml has been modified
found difference in CALENDAR : case 'GREGORIAN' locked 'NO_LEAP'
Setting build complete to False

[wwieder]

I confirmed that this works in dev171

./run_neon --neon-sites MOAB --overwrite --experiment dev171_noRuntype --output-root /glade/derecho/scratch/wwieder/neon_AK

The base case uses NO_LEAP and the .transient case uses GREGORIAN

[wwieder]

forcing the base case in dev175 works~

$./run_neon --neon-sites TALL --run-type transient --experiment dev175 --output-root /glade/derecho/scratch/wwieder/neon_AK --overwrite

now the base case and transient case are both Gregorian

[wwieder]

now we still need to set up base cases and for ad and postad that are no_leap