Merge branch 'feature/initialization_improvements'

Reuse some files generated in initialization when rerunning

A new directory is now in your run directory init_generated_files. These
two run time generated files - finidat_interp_dest.nc and the land
fraction file ctsm_landfrac.nc (which is new in this PR) - are put in
that directory when they are created. Then, if they already exist, they
are reused rather than being regenerated. (Technically, it looks for a
status file that flags successful creation of this file.) This applies
when you rerun a startup (non-restart) run a second or subsequent time,
as is common when doing development and testing.

For tests, theinit_generated_files directory is removed before running,
so that rerunning a test will do the same thing the second time.

There are occasional, rare cases when a user would need to manually
remove the init_generated_files directory – e.g., when changing things
about N dominant landunits or PFTs.

Also, output was cleaned up in a few cases so that it did not span
multiple lines.

Externals.cfg

@@ -44,7 +44,7 @@ required = True
 local_path = cime
 protocol = git
 repo_url = https://github.com/ESMCI/cime
-tag = cime6.0.40
+tag = cime6.0.45
 required = True
 
 [cmeps]

bld/namelist_files/namelist_definition_ctsm.xml

@@ -8,16 +8,6 @@
 <!-- CLM Namelist -->
 <!-- ========================================================================================  -->
 
-<entry id="finidat_interp_dest" 
-       type="char*256" 
-       category="datasets"
-       input_pathname="abs" 
-       group="clm_inparm" 
-       valid_values="" >
-If use_init_interp is set to .true., interpinic will be called to interpolate
-the file given by finidat, creating the output file specified by finidat_interp_dest.
-</entry>
-
 <entry id="finidat" 
        type="char*256" 
        category="datasets"

cime_config/SystemTests/lilacsmoke.py

@@ -217,6 +217,10 @@ def _setup_atm_driver_rundir(self):
             symlink_force(os.path.join(self._runtime_inputs_dir(), file_to_link),
                           os.path.join(rundir, file_to_link))
 
+        init_generated_files_dir = os.path.join(rundir, "init_generated_files")
+        if not os.path.exists(init_generated_files_dir):
+            os.mkdir(init_generated_files_dir)
+
     def _cmpgen_namelists(self):
         """Redoes the namelist comparison & generation with appropriate namelists
 

cime_config/buildnml

@@ -62,6 +62,11 @@ def buildnml(case, caseroot, compname):
     mask = case.get_value("MASK_GRID")
     driver = case.get_value("COMP_INTERFACE").lower()
 
+    # Create init_generated_files directory if not there
+    newdir = os.path.join(rundir,"init_generated_files")
+    if not os.path.exists(newdir):
+        os.mkdir(newdir)
+
     # -----------------------------------------------------
     # Error checking
     # -----------------------------------------------------

cime_config/testdefs/testlist_clm.xml

@@ -2532,6 +2532,17 @@
     </machines>
   </test>
 
+  <test name="REUSEINITFILES_D_Ld1" grid="f10_f10_mg37" compset="I1850Clm50BgcCrop" testmods="clm/default">
+    <machines>
+      <machine name="cheyenne" compiler="gnu" category="aux_clm">
+        <options>
+          <option name="wallclock">0:20:00</option>
+          <option name="comment">Make sure that a rerun with already-generated initialization files is bit-for-bit</option>
+        </options>
+      </machine>
+    </machines>
+  </test>
+
   <test name="FUNITCTSM_P1x1" grid="f10_f10_mg37" compset="I2000Clm50Sp">
     <machines>
       <machine name="cheyenne" compiler="intel" category="aux_clm">

doc/ChangeLog

@@ -1,4 +1,110 @@
 ===============================================================
+Tag name: ctsm5.1.dev106
+Originator(s): mvertens (Mariana Vertenstein), sacks (Bill Sacks)
+Date: Thu Aug  4 15:11:31 MDT 2022
+One-line Summary: Reuse some files generated in initialization when rerunning
+
+Purpose and description of changes
+----------------------------------
+
+A new directory is now in your run directory init_generated_files. These
+two run time generated files - finidat_interp_dest.nc and the land
+fraction file ctsm_landfrac.nc (which is new in this PR) - are put in
+that directory when they are created. Then, if they already exist, they
+are reused rather than being regenerated. (Technically, it looks for a
+status file that flags successful creation of this file.) This applies
+when you rerun a startup (non-restart) run a second or subsequent time,
+as is common when doing development and testing.
+
+For tests, theinit_generated_files directory is removed before running,
+so that rerunning a test will do the same thing the second time.
+
+There are occasional, rare cases when a user would need to manually
+remove the init_generated_files directory – e.g., when changing things
+about N dominant landunits or PFTs.
+
+Also, output was cleaned up in a few cases so that it did not span
+multiple lines.
+
+Significant changes to scientifically-supported configurations
+--------------------------------------------------------------
+
+Does this tag change answers significantly for any of the following physics configurations?
+(Details of any changes will be given in the "Answer changes" section below.)
+
+    [Put an [X] in the box for any configuration with significant answer changes.]
+
+[ ] clm5_1
+
+[ ] clm5_0
+
+[ ] ctsm5_0-nwp
+
+[ ] clm4_5
+
+
+Notes of particular relevance for users
+---------------------------------------
+Caveats for users (e.g., need to interpolate initial conditions):
+- If you do a run, then change a runtime parameter that makes the
+  previous finidat_interp_dest.nc file incompatible with the new
+  configuration (e.g., changing N dominant landunits or PFTs) then
+  rerun, you will need to remove the contents of the
+  run/init_generated_files subdirectory before rerunning. (You do *not*
+  need to remove this directory when changing finidat in user_nl_clm: in
+  this specific case, CTSM detects the change and regenerates
+  finidat_interp_dest even if one already exists.)
+
+Changes to CTSM's user interface (e.g., new/renamed XML or namelist variables):
+- Removed finidat_interp_dest from the namelist; it is now hard-coded to
+  finidat_interp_dest.nc. (This seemed not to be used much, if ever, and
+  supporting this would have complicated the logic in this PR.)
+
+Notes of particular relevance for developers:
+---------------------------------------------
+Changes to tests or testing:
+- Added a new test using a new test type (defined in CIME):
+  REUSEINITFILES. This runs two cases and compares them: the first runs
+  a case as normal; the second copies in the init_generated_files
+  directory from case one and then does a run that uses these saved
+  init_generated_files. This ensures that using the init_generated_files
+  does not change answers.
+
+
+Testing summary:
+----------------
+
+  regular tests (aux_clm: https://github.com/ESCOMP/CTSM/wiki/System-Testing-Guide#pre-merge-system-testing):
+
+    cheyenne ---- PASS
+    izumi ------- PASS
+
+    Two tests failed initially due to what appeared to be system issues:
+    - ERP_P36x2_D_Ld5.f10_f10_mg37.I1850Clm45BgcCru.cheyenne_intel.clm-default
+      (runtime error initially, passed upon rerun)
+    - SMS_Ly5_Mmpi-serial.1x1_brazil.IHistClm50BgcQianRs.izumi_intel.clm-newton_krylov_spinup 
+      (build failure initially; passed after removing the build
+      directory, rebuilding and rerunning)
+
+Answer changes
+--------------
+
+Changes answers relative to baseline: NO
+
+    Most FATES tests had baseline comparison failures, just for the
+    field fates_levcdam, but this turned out to be due to an error in
+    the stored baselines for ctsm5.1.dev104 that has now been corrected.
+
+Other details
+-------------
+List any externals directories updated (cime, rtm, mosart, cism, fates, etc.):
+- cime: cime6.0.40 -> cime6.0.45
+
+Pull Requests that document the changes (include PR ids):
+https://github.com/ESCOMP/CTSM/pull/1793
+
+===============================================================
+===============================================================
 Tag name: ctsm5.1.dev105
 Originator(s): slevis (Samuel Levis,SLevis Consulting,303-665-1310)
 Date: Tue Jul 26 23:57:31 MDT 2022

doc/ChangeSum

@@ -1,5 +1,6 @@
 Tag                   Who      Date  Summary
 ============================================================================================================================
+    ctsm5.1.dev106 multiple 08/04/2022 Reuse some files generated in initialization when rerunning
     ctsm5.1.dev105   slevis 07/26/2022 make interfaces more similar between fsurdat_modifier and subset_data tools
     ctsm5.1.dev104 glemieux 07/15/2022 Update to FATES history names and machine configuration
     ctsm5.1.dev103    sacks 07/12/2022 Fix accumulation variables when changing model time step

doc/source/lilac/obtaining-building-and-running/setting-ctsm-runtime-options.rst

@@ -51,6 +51,8 @@ also illustrated in :numref:`Figure ctsm_lilac_runtime_file_workflow`:
 #. Copy or create links to ``lnd_in``, ``lnd_modelio.nml`` and ``lilac_in`` in the
    directory from which you will be running the model.
 
+#. Create a subdirectory named ``init_generated_files`` inside the directory from which you will be running the model.
+
 .. _Figure ctsm_lilac_runtime_file_workflow:
 
 .. figure:: ctsm_lilac_runtime_file_workflow.*
@@ -230,8 +232,7 @@ files (and not to CTSM's standard input files), then this is nothing to worry ab
 Copying the necessary files to the model's run directory
 ========================================================
 
-Finally, copy or create links to the following files in the directory from which you will
-run the model:
+Copy or create links to the following files in the directory from which you will run the model:
 
 - ``lnd_in``: This is the main namelist input file for CTSM
 
@@ -263,3 +264,8 @@ run the model:
      machine. The default for this variable is controlled by the ``--no-pnetcdf`` argument
      to ``build_ctsm``, but you can change it here if you mistakenly set or didn't set
      ``--no-pnetcdf`` when running ``build_ctsm``.
+
+Creating an init_generated_files directory
+==========================================
+
+Finally, from the directory where you will run the model, create a subdirectory named ``init_generated_files``. This directory is used to store some files generated by CTSM in model initialization which are reused if you redo a startup (i.e., non-restart) run from the same directory. (In some situations, such as changing CTSM's ``finidat`` file, you will need to remove this directory before rerunning. CTSM attempts to detect these situations and notify you if this is needed.)

src/cpl/nuopc/lnd_import_export.F90

@@ -663,7 +663,7 @@ subroutine import_fields( gcomp, bounds, glc_present, rof_prognostic, &
 
     ! Atmosphere ndep
     if (fldchk(importState, Faxa_ndep)) then
-       ! The mediator is sending ndep in units if kgN/m2/s - and ctsm
+       ! The mediator is sending ndep in units of kgN/m2/s - and ctsm
        ! uses units of gN/m2/sec so the following conversion needs to happen
        call state_getimport_2d(importState, Faxa_ndep, forc_ndep(begg:,:), rc=rc)
        if (ChkErr(rc,__LINE__,u_FILE_u)) return