Automate creation of symlinks to fixed files (i.e. pregenerated grid,…

… orography, and surface climatology files) in NCO mode; treat fixed files in NCO mode more in line with that in community mode

## DESCRIPTION OF CHANGES:

### Main changes:
* In NCO mode, remove requirement that there be "fix_am" and "fix_lam" symlinks under a directory named "fix" under "regional_workflow".  Instead, move "fix_am" and "fix_lam" to under the experiment directory, and have "fix_am" be a symlink like before but (for consistency with what is done in community mode) have "fix_lam" be an actual directory (not a symlink) that contains symlinks to files in some other specified directory.  
  * For clarity, move creation of the "fix_am" symlink in NCO mode from setup.sh to generate_FV3LAM_wflow.sh, and make it part of the same if-statement in generate_FV3LAM_wflow.sh that creates the "fix_am" directory in community mode.
  * Assume that in NCO mode, the pregenerated grid, orography, and surface climatology files are located in the directory ${FIXLAM_NCO_BASEDIR}/${PREDEF_GRID_NAME}, where FIXLAM_NCO_BASEDIR is a new workflow variable that specifies the base directory in which these files are located.  Thus, the files must be located in a subdirectory having the name of the predefined grid under ${FIXLAM_NCO_BASEDIR}.  Note that FIXLAM_NCO_BASEDIR is only used in NCO mode.
  * With the above changes, the creation of symlinks in the "fix_lam" subdirectory under the experiment directory and the setting of the variable RES_IN_FIXLAM_FILENAMES can now be handled in the same way in NCO mode as in community mode.  Thus, remove section of code in setup.sh that sets RES_IN_FIXLAM_FILENAMES specifically in NCO mode, and use the section of code that was previously only for community mode to create the symlinks and set RES_IN_FIXLAM_FILENAMES in both NCO and community modes.
* Remove check near the top of setup.sh on existence of SFC_CLIMO_DIR when RUN_TASK_MAKE_SFC_CLIMO is set to "FALSE" because this test is in effect now performed later in the script, both for NCO mode and community mode.
* For simplicity, remove setting of SFC_CLIMO_DIR to a null string when it is not already a null string and RUN_TASK_MAKE_SFC_CLIMO is set to "TRUE" because this is not done for GRID_DIR and OROG_DIR.
* For the purposes of running the WE2E tests, assume that when using pregenerated grid, orography, and surface climatology files in community mode, the files are located directly under the ${PREDEF_GRID_NAME} subdirectory of some specified base directory just as we assume is the case in NCO mode.  Thus, in run_experiments.sh, remove the "grid", "orog", and "sfc_climo" subdirectories in the settings of GRID_DIR, OROG_DIR, and SFC_CLIMO_DIR.  Note that this is only for the WE2E testing system.  Users can still specify (in config.sh) independent directories for GRID_DIR, OROG_DIR, and SFC_CLIMO_DIR when using pregenerated files in community mode.
* In the sample NCO mode workflow configuration file config.nco.sh, added the variable FIXLAM_NCO_BASEDIR (for a valid value on Hera only; values are platform and user dependent).

### Improvements:
* Add missing code that checks that RUN_TASK_MAKE_OROG is set to a valid value (just like there is code to check that RUN_TASK_MAKE_GRID and RUN_TASK_MAKE_SFC_CLIMO are set to valid values).
* Edit error messages in setup.sh to make them more accurate and clearer.
* Clean up comments.

## TESTS CONDUCTED: 
Ran the following WE2E tests on Hera:
* nco_CONUS_25km_GFDLgrid
* nco_RRFS_CONUS_25km_HRRR_RAP
* pregen_grid_orog_sfc_climo
* regional_002

These are ones that test the workflow in NCO mode (nco_CONUS_25km_GFDLgrid, nco_RRFS_CONUS_25km_HRRR_RAP), in community mode with pregenerated files (pregen_grid_orog_sfc_climo), and in community mode without pregenerated files (regional_002).  All tests were successful.

jobs/JREGIONAL_MAKE_GRID

@@ -13,8 +13,8 @@
 # 1) grid_gen_scr:
 #
 #    This script generates grid files that will be used by subsequent
-#    preprocessing steps.  It places its output in the temporary direc-
-#    tory defined in GRID_DIR.  Note that:
+#    preprocessing steps.  It places its output in the directory defined 
+#    by GRID_DIR.  Note that:
 #
 #    a) This script creates grid files for each of the 7 tiles of the
 #       cubed sphere grid (where tiles 1 through 6 cover the globe, and
@@ -44,7 +44,7 @@
 # 2) orog_gen_scr:
 #
 #    This script generates the orography file.  It places its output in
-#    the temporary directory defined in OROG_DIR.  Note that:
+#    the directory defined by OROG_DIR.  Note that:
 #
 #    a) This script generates an orography file only on tile 7.
 #

scripts/exregional_make_orog.sh

@@ -350,7 +350,6 @@ fn_suffix_with_halo="tile${TILE_RGNL}.halo${NHW}.nc"
 raw_orog_fn="${raw_orog_fn_prefix}.${fn_suffix_with_halo}"
 raw_orog_fp="${raw_dir}/${raw_orog_fn}"
 mv_vrfy "${raw_orog_fp_orig}" "${raw_orog_fp}"
-
 #
 #-----------------------------------------------------------------------
 #

tests/run_experiments.sh

@@ -493,6 +493,13 @@ VERBOSE=\"${VERBOSE}\""
      [ ${RUN_TASK_MAKE_OROG} = "FALSE" ] || \
      [ ${RUN_TASK_MAKE_SFC_CLIMO} = "FALSE" ]; then
 
+# Note:
+# Now that the "grid", "orog", and "sfc_climo" sub-subdirectories under
+# pregen_basedir have been removed, we don't need the variable pregen_basedir
+# and can instead have the variable "pregen_dir" that gets set to 
+# ${pregen_basedir}/${PREDEF_GRID_NAME}, and pregen_dir can then be used
+# to set GRID_DIR, OROG_DIR, and/or SFC_CLIMO_DIR below.
+
     if [ "$MACHINE" = "HERA" ]; then
       pregen_basedir="/scratch2/BMC/det/FV3LAM_pregen"
     elif [ "$MACHINE" = "CHEYENNE" ]; then
@@ -510,19 +517,7 @@ specified for this machine (MACHINE):
 # Directory for pregenerated grid files.
 #
   if [ ${RUN_TASK_MAKE_GRID} = "FALSE" ]; then
-
-    GRID_DIR="${pregen_basedir}/grid/${PREDEF_GRID_NAME}"
-    if [ "$MACHINE" = "HERA" ]; then
-      GRID_DIR="/scratch2/BMC/det/FV3LAM_pregen/grid/${PREDEF_GRID_NAME}"
-    elif [ "$MACHINE" = "CHEYENNE" ]; then
-      GRID_DIR="/glade/p/ral/jntp/UFS_CAM/FV3LAM_pregen/grid/${PREDEF_GRID_NAME}"
-    else
-      print_err_msg_exit "\
-The directory (GRID_DIR) in which the pregenerated grid files are located
-has not been specified for this machine (MACHINE):
-  MACHINE= \"${MACHINE}\""
-    fi
-
+    GRID_DIR="${pregen_basedir}/${PREDEF_GRID_NAME}"
     str=${str}"
 #
 # Directory containing the pregenerated grid files.
@@ -534,18 +529,7 @@ GRID_DIR=\"${GRID_DIR}\""
 # Directory for pregenerated orography files.
 #
   if [ ${RUN_TASK_MAKE_OROG} = "FALSE" ]; then
-
-    if [ "$MACHINE" = "HERA" ]; then
-      OROG_DIR="/scratch2/BMC/det/FV3LAM_pregen/orog/${PREDEF_GRID_NAME}"
-    elif [ "$MACHINE" = "CHEYENNE" ]; then
-      OROG_DIR="/glade/p/ral/jntp/UFS_CAM/FV3LAM_pregen/orog/${PREDEF_GRID_NAME}"
-    else
-      print_err_msg_exit "\
-The directory (OROG_DIR) in which the pregenerated grid files are located
-has not been specified for this machine (MACHINE):
-  MACHINE= \"${MACHINE}\""
-    fi
-
+    OROG_DIR="${pregen_basedir}/${PREDEF_GRID_NAME}"
     str=${str}"
 #
 # Directory containing the pregenerated orography files.
@@ -557,18 +541,7 @@ OROG_DIR=\"${OROG_DIR}\""
 # Directory for pregenerated surface climatology files.
 #
   if [ ${RUN_TASK_MAKE_SFC_CLIMO} = "FALSE" ]; then
-
-    if [ "$MACHINE" = "HERA" ]; then
-      SFC_CLIMO_DIR="/scratch2/BMC/det/FV3LAM_pregen/sfc_climo/${PREDEF_GRID_NAME}"
-    elif [ "$MACHINE" = "CHEYENNE" ]; then
-      SFC_CLIMO_DIR="/glade/p/ral/jntp/UFS_CAM/FV3LAM_pregen/sfc_climo/${PREDEF_GRID_NAME}"
-    else
-      print_err_msg_exit "\
-The directory (SFC_CLIMO_DIR) in which the pregenerated grid files are
-located has not been specified for this machine (MACHINE):
-  MACHINE= \"${MACHINE}\""
-    fi
-
+    SFC_CLIMO_DIR="${pregen_basedir}/${PREDEF_GRID_NAME}"
     str=${str}"
 #
 # Directory containing the pregenerated surface climatology files.
@@ -628,21 +601,7 @@ SFC_CLIMO_DIR=\"${SFC_CLIMO_DIR}\""
 #   \$PTMP/com/\$NET/\$RUN/\$RUN.\$yyyymmdd/\$hh
 #
 RUN=\"\${EXPT_SUBDIR}\"
-envir=\"\${EXPT_SUBDIR}\"
-#
-# In NCO mode, the user must manually (e.g. after doing the build step)
-# create the symlink \"\${FIXrrfs}/fix_sar\" that points to EMC's FIXsar
-# directory on the machine.  For example, on hera, the symlink's target
-# needs to be
-#
-#   /scratch2/NCEPDEV/fv3-cam/emc.campara/fix_fv3cam/fix_sar
-#
-# The experiment generation script will then set FIXsar to
-#
-#   FIXsar=\"\${FIXrrfs}/fix_sar/\${PREDEF_GRID_NAME}\"
-#
-# where PREDEF_GRID_NAME has the value set above.
-#"
+envir=\"\${EXPT_SUBDIR}\""
 #
 # Set COMINgfs.
 #
@@ -653,10 +612,13 @@ envir=\"\${EXPT_SUBDIR}\"
 
       if [ "$MACHINE" = "HERA" ]; then
         COMINgfs="/scratch1/NCEPDEV/hwrf/noscrub/hafs-input/COMGFS"
+        FIXLAM_NCO_BASEDIR="/scratch2/BMC/det/FV3LAM_pregen"
       elif [ "$MACHINE" = "JET" ]; then
         COMINgfs="/lfs1/HFIP/hwrf-data/hafs-input/COMGFS"
+        FIXLAM_NCO_BASEDIR="/needs/to/be/specified"
       elif [ "$MACHINE" = "CHEYENNE" ]; then
         COMINgfs="/glade/scratch/ketefian/NCO_dirs/COMGFS"
+        FIXLAM_NCO_BASEDIR="/needs/to/be/specified"
       else
         print_err_msg_exit "\
 The directory (COMINgfs) that needs to be specified when running the
@@ -672,7 +634,15 @@ for this machine (MACHINE):
 # mode (RUN_ENVIR set to \"nco\") AND using the FV3GFS or the GSMGFS as
 # the external model for ICs and/or LBCs.
 #
-COMINgfs=\"${COMINgfs}\""
+COMINgfs=\"${COMINgfs}\"
+#
+# The base directory in which the pregenerated grid, orography, and surface 
+# climatology \"fixed\" files used in NCO mode are located.  In NCO mode,
+# the workflow scripts will create a symlink (at the location specified 
+# by FIXLAM) to a subdirectory under FIXLAM_NCO_BASDEDIR.  (The name of 
+# the subdirectory is the name of the grid specified by PREDEF_GRID_NAME.)
+#
+FIXLAM_NCO_BASEDIR=\"${FIXLAM_NCO_BASEDIR}\""
 
     fi
 #

ush/config.nco.sh

@@ -22,24 +22,12 @@ CYCL_HRS=( "18" )
 EXTRN_MDL_NAME_ICS="FV3GFS"
 EXTRN_MDL_NAME_LBCS="FV3GFS"
 
+#
 # The following must be modified for different platforms and users.
+#
 RUN="an_experiment"
 COMINgfs="/scratch1/NCEPDEV/hwrf/noscrub/hafs-input/COMGFS"     # Path to directory containing files from the external model (FV3GFS).
+FIXLAM_NCO_BASEDIR="/scratch2/BMC/det/FV3LAM_pregen"            # Path to directory containing the pregenerated grid, orography, and surface climatology "fixed" files to use for the experiment.
 STMP="/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/NCO_dirs/stmp"  # Path to directory STMP that mostly contains input files.
 PTMP="/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/NCO_dirs/ptmp"  # Path to directory PTMP in which the experiment's output files will be placed.
 
-#
-# In NCO mode, the user must manually (e.g. after doing the build step)
-# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
-# directory on the machine.  For example, on hera, the symlink's target
-# needs to be
-#
-#   /scratch2/NCEPDEV/fv3-cam/emc.campara/fix_fv3cam/fix_sar
-#
-# The experiment generation script will then set FIXLAM to 
-#
-#   FIXLAM="${FIXrrfs}/fix_lam/${PREDEF_GRID_NAME}"
-#
-# where PREDEF_GRID_NAME has the value set above.
-#
-

ush/config_defaults.sh

@@ -175,6 +175,19 @@ EXPT_SUBDIR=""
 #
 #   $COMINgfs/gfs.$yyyymmdd/$hh
 #
+# FIXLAM_NCO_BASEDIR:
+# The base directory containing pregenerated grid, orography, and surface 
+# climatology files.  For the pregenerated grid specified by PREDEF_GRID_NAME, 
+# these "fixed" files are located in:
+#
+#   ${FIXLAM_NCO_BASEDIR}/${PREDEF_GRID_NAME}
+#
+# The workflow scripts will create a symlink in the experiment directory
+# that will point to a subdirectory (having the name of the grid being
+# used) under this directory.  This variable should be set to a null 
+# string in this file, but it can be specified in the user-specified 
+# workflow configuration file (EXPT_CONFIG_FN)
+#
 # STMP:
 # The beginning portion of the directory that will contain cycle-dependent
 # model input files, symlinks to cycle-independent input files, and raw 
@@ -214,6 +227,7 @@ EXPT_SUBDIR=""
 #-----------------------------------------------------------------------
 #
 COMINgfs="/base/path/of/directory/containing/gfs/input/files"
+FIXLAM_NCO_BASEDIR=""
 STMP="/base/path/of/directory/containing/model/input/and/raw/output/files"
 NET="rrfs"
 envir="para"

ush/generate_FV3LAM_wflow.sh

@@ -407,16 +407,39 @@ fi
 #
 #-----------------------------------------------------------------------
 #
-# Copy fixed files from system directory to the FIXam directory (which
-# is under the experiment directory).  Note that some of these files get
-# renamed during the copy process.
+# Create the FIXam directory under the experiment directory.  In NCO mode,
+# this will be a symlink to the directory specified in FIXgsm, while in
+# community mode, it will be an actual directory with files copied into
+# it from FIXgsm.
 #
 #-----------------------------------------------------------------------
 #
+# First, consider NCO mode.
+#
+if [ "${RUN_ENVIR}" = "nco" ]; then
 
-# In NCO mode, we assume the following copy operation is done beforehand,
-# but that can be changed.
-if [ "${RUN_ENVIR}" != "nco" ]; then
+  ln_vrfy -fsn "$FIXgsm" "$FIXam"
+#
+# Resolve the target directory that the FIXam symlink points to and check 
+# that it exists.
+#
+  path_resolved=$( readlink -m "$FIXam" )
+  if [ ! -d "${path_resolved}" ]; then
+    print_err_msg_exit "\
+In order to be able to generate a forecast experiment in NCO mode (i.e.
+when RUN_ENVIR set to \"nco\"), the path specified by FIXam after resolving
+all symlinks (path_resolved) must be an existing directory (but in this
+case isn't):
+  RUN_ENVIR = \"${RUN_ENVIR}\"
+  FIXam = \"$FIXam\"
+  path_resolved = \"${path_resolved}\"
+Please ensure that path_resolved is an existing directory and then rerun
+the experiment generation script."
+  fi
+#
+# Now consider community mode.
+#
+else
 
   print_info_msg "$VERBOSE" "
 Copying fixed files from system directory (FIXgsm) to a subdirectory