Add SPP to the SRW App (#685)

* Add default namelist with SPP entries.

* Changes necessary to run with SPP

* Typo fix in generate script.

* Changes to implement SPP.

* Add comment regarding use of SPP.

* Pass N_VAR_SPP to the var_defns.sh file.

* Add spp_wts_pbl to the FV3_GSD_SAR SDF diag_table file

* Remove the contents of the &nam_spperts stanza and n_var_spp from the namelist when not using SPP.

* Remove SPP namelist entries from the template input.nml file.  These values are now handled in the generate script if using SPP.

* Add MYNN SFC pattern variable to the diag_table file for the FV3_GSD_SAR SDF

* Add 'sfc' perturbation option to SPP

* Add iseed array to namelist generation

* Changes to add "rad" and "gwd" SPP perturbations.

* Add SPP and ad-hoc stochastic physics to SDFs.

* Add LSM SPP functionality to the SRW App.

* Add random number generation for LSM SPP iseeds in ensemble mode.

* Fix undeclared variable for LSM SPP.

* Add if statement for LSM SPP namelist entries and set fhcyc to 999 if LSM SPP is turned on.

* Modify how namelist settings are applied for LSM SPP.

* Fix implementation of fhcyc changes.

* Typo fix.

* Add do_gsl* namelist entries to YAML file for FV3_HRRR SDF.

* Remove diss_est from the diag table files since it's unavailable for now.

* Change LSM SPP perturbation seed to be the same as all other SPP.

* Changes to FV3_HRRR field_table

* Change to the FV3_HRRR field_table file

* Update name for SPP block in input.nml

* Change SPP_LSM_* to LSM_SPP_*

* Remove space.

* Update descriptions of LSM perturbations.

* Shut off PET file generation.

* Modifications for land and SPP perturbations (templates, namelists, default values, etc.)

* Remove diss_est from diag_table files and remove the FV3_GSD_SAR and FV3_GSD_v0 diag_table files.

* Fix in setup.sh for LSM SPP

* Update Thompson MP SPP settings.

* Add back FV3_GSD_SAR and FV3_GSD_v0 SDFs.  Will be removed in future, separate PR.

* Requested modifications to the input.nml template and in-line documentation changes.

* Fix check for LSM SPP namelist settings

* Fix to LSM SPP namelist check.

* Changes request based on PR review.

* Changes requested in PR review.

* Variable descriptions.

* Fix comment formatting.

* Fix MET/METplus/obs paths in machine files to allow for user-defined settings in config.sh

* Only add/modify stochastic physics namelist entries when running with SPP, LSM SPP, SPPT, SHUM, or SKEB.

ush/config_defaults.sh

@@ -507,78 +507,78 @@ WRITE_DOPOST="FALSE"
 #
 # CCPA_OBS_DIR:
 # User-specified location of top-level directory where CCPA hourly
-# precipitation files used by METplus are located. This parameter needs
-# to be set for both user-provided observations and for observations 
-# that are retrieved from the NOAA HPSS (if the user has access) via
-# the get_obs_ccpa_tn task (activated in workflow by setting 
+# precipitation files used by METplus are located. This parameter needs
+# to be set for both user-provided observations and for observations 
+# that are retrieved from the NOAA HPSS (if the user has access) via
+# the get_obs_ccpa_tn task (activated in workflow by setting
 # RUN_TASK_GET_OBS_CCPA="TRUE"). In the case of pulling observations 
 # directly from NOAA HPSS, the data retrieved will be placed in this 
 # directory. Please note, this path must be defind as 
 # /full-path-to-obs/ccpa/proc. METplus is configured to verify 01-, 
-# 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files. 
+# 03-, 06-, and 24-h accumulated precipitation using hourly CCPA files.
 # METplus configuration files require the use of predetermined directory 
 # structure and file names. Therefore, if the CCPA files are user 
 # provided, they need to follow the anticipated naming structure: 
-# {YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2, where YYYY is the 4-digit 
-# valid year, MM the 2-digit valid month, DD the 2-digit valid day of 
-# the month, and HH the 2-digit valid hour of the day. In addition, a 
+# {YYYYMMDD}/ccpa.t{HH}z.01h.hrap.conus.gb2, where YYYY is the 4-digit 
+# valid year, MM the 2-digit valid month, DD the 2-digit valid day of 
+# the month, and HH the 2-digit valid hour of the day. In addition, a 
 # caveat is noted for using hourly CCPA data. There is a problem with 
 # the valid time in the metadata for files valid from 19 - 00 UTC (or 
-# files  under the '00' directory). The script to pull the CCPA data 
+# files under the '00' directory). The script to pull the CCPA data 
 # from the NOAA HPSS has an example of how to account for this as well
-# as organizing the data into a more intuitive format: 
+# as organizing the data into a more intuitive format: 
 # regional_workflow/scripts/exregional_get_ccpa_files.sh. When a fix
 # is provided, it will be accounted for in the
 # exregional_get_ccpa_files.sh script.
 #
 # MRMS_OBS_DIR:
-# User-specified location of top-level directory where MRMS composite
-# reflectivity files used by METplus are located. This parameter needs
-# to be set for both user-provided observations and for observations
-# that are retrieved from the NOAA HPSS (if the user has access) via the
-# get_obs_mrms_tn task (activated in workflow by setting 
-# RUN_TASK_GET_OBS_MRMS="TRUE"). In the case of pulling observations 
+# User-specified location of top-level directory where MRMS composite
+# reflectivity files used by METplus are located.  This parameter needs
+# to be set for both user-provided observations and for observations
+# that are retrieved from the NOAA HPSS (if the user has access) via the
+# get_obs_mrms_tn task (activated in workflow by setting
+# RUN_TASK_GET_OBS_MRMS="TRUE").  In the case of pulling observations 
 # directly from NOAA HPSS, the data retrieved will be placed in this 
 # directory. Please note, this path must be defind as 
 # /full-path-to-obs/mrms/proc. METplus configuration files require the
-# use of predetermined directory structure and file names. Therefore, if
-# the MRMS files are user provided, they need to follow the anticipated 
+# use of predetermined directory structure and file names. Therefore, if
+# the MRMS files are user provided, they need to follow the anticipated 
 # naming structure:
-# {YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2,
-# where YYYY is the 4-digit valid year, MM the 2-digit valid month, DD 
-# the 2-digit valid day of the month, HH the 2-digit valid hour of the 
+# {YYYYMMDD}/MergedReflectivityQCComposite_00.50_{YYYYMMDD}-{HH}{mm}{SS}.grib2,
+# where YYYY is the 4-digit valid year, MM the 2-digit valid month, DD 
+# the 2-digit valid day of the month, HH the 2-digit valid hour of the 
 # day, mm the 2-digit valid minutes of the hour, and SS is the two-digit
 # valid seconds of the hour. In addition, METplus is configured to look
 # for a MRMS composite reflectivity file for the valid time of the 
 # forecast being verified; since MRMS composite reflectivity files do 
 # not always exactly match the valid time, a script, within the main 
 # script to retrieve MRMS data from the NOAA HPSS, is used to identify
-# and rename the MRMS composite reflectivity file to match the valid
-# time of the forecast. The script to pull the MRMS data from the NOAA 
+# and rename the MRMS composite reflectivity file to match the valid
+# time of the forecast.  The script to pull the MRMS data from the NOAA 
 # HPSS has an example of the expected file naming structure: 
 # regional_workflow/scripts/exregional_get_mrms_files.sh. This script 
 # calls the script used to identify the MRMS file closest to the valid 
 # time: regional_workflow/ush/mrms_pull_topofhour.py.
 #
 # NDAS_OBS_DIR:
 # User-specified location of top-level directory where NDAS prepbufr 
-# files used by METplus are located. This parameter needs to be set for
-# both user-provided observations and for observations that are 
-# retrieved from the NOAA HPSS (if the user has access) via the 
+# files used by METplus are located. This parameter needs to be set for
+# both user-provided observations and for observations that are 
+# retrieved from the NOAA HPSS (if the user has access) via the 
 # get_obs_ndas_tn task (activated in workflow by setting 
 # RUN_TASK_GET_OBS_NDAS="TRUE"). In the case of pulling observations 
 # directly from NOAA HPSS, the data retrieved will be placed in this 
 # directory. Please note, this path must be defind as 
 # /full-path-to-obs/ndas/proc. METplus is configured to verify 
 # near-surface variables hourly and upper-air variables at times valid 
-# at 00 and 12 UTC with NDAS prepbufr files. METplus configuration files
-# require the use of predetermined file names. Therefore, if the NDAS 
-# files are user provided, they need to follow the anticipated naming 
-# structure: prepbufr.ndas.{YYYYMMDDHH}, where YYYY is the 4-digit valid
-# year, MM the 2-digit valid month, DD the 2-digit valid day of the 
-# month, and HH the 2-digit valid hour of the day. The script to pull 
+# at 00 and 12 UTC with NDAS prepbufr files.  METplus configuration files
+# require the use of predetermined file names. Therefore, if the NDAS 
+# files are user provided, they need to follow the anticipated naming 
+# structure: prepbufr.ndas.{YYYYMMDDHH}, where YYYY is the 4-digit valid
+# year, MM the 2-digit valid month, DD the 2-digit valid day of the 
+# month, and HH the 2-digit valid hour of the day. The script to pull 
 # the NDAS data from the NOAA HPSS has an example of how to rename the
-# NDAS data into a more intuitive format with the valid time listed in 
+# NDAS data into a more intuitive format with the valid time listed in 
 # the file name: regional_workflow/scripts/exregional_get_ndas_files.sh
 #
 #-----------------------------------------------------------------------
@@ -1505,6 +1505,7 @@ FIXgsm_FILES_TO_COPY_TO_FIXam=( \
 "global_zorclim.1x1.grb" \
 "global_sfc_emissivity_idx.txt" \
 "global_solarconstant_noaa_an.txt" \
+"global_albedo4.1x1.grb" \
 "geo_em.d01.lat-lon.2.5m.HGT_M.nc" \
 "HGT.Beljaars_filtered.lat-lon.30s_res.nc" \
 "replace_with_FIXgsm_ozone_prodloss_filename" \
@@ -1555,6 +1556,7 @@ CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING=( \
 "co2historicaldata_glob.txt | global_co2historicaldata_glob.txt" \
 "co2monthlycyc.txt          | co2monthlycyc.txt" \
 "global_h2oprdlos.f77       | global_h2o_pltc.f77" \
+"global_albedo4.1x1.grb     | global_albedo4.1x1.grb" \
 "global_zorclim.1x1.grb     | global_zorclim.1x1.grb" \
 "sfc_emissivity_idx.txt     | global_sfc_emissivity_idx.txt" \
 "solarconstant_noaa_an.txt  | global_solarconstant_noaa_an.txt" \
@@ -1808,27 +1810,62 @@ USE_ZMTNBLCK="FALSE"
 #
 #-----------------------------------------------------------------------
 #
-# Set default SPP stochastic physics options.
-# Each SPP option is an array, applicable (in order) to the scheme/parameter
-# listed in SPP_VAR_LIST. Enter each value of the array in config.sh as 
-# shown below without commas or single quotes (e.g., SPP_VAR_LIST=
-# ( "pbl" "lsm" "mp" ). Both commas and single quotes will be added by
-# Jinja when creating the namelist.
+# Set default SPP stochastic physics options. Each SPP option is an array, 
+# applicable (in order) to the scheme/parameter listed in SPP_VAR_LIST. 
+# Enter each value of the array in config.sh as shown below without commas
+# or single quotes (e.g., SPP_VAR_LIST=( "pbl" "sfc" "mp" "rad" "gwd" ). 
+# Both commas and single quotes will be added by Jinja when creating the
+# namelist.
 #
 # Note that SPP is currently only available for specific physics schemes 
 # used in the RAP/HRRR physics suite.  Users need to be aware of which SDF
 # is chosen when turning this option on. 
 #
+# Patterns evolve and are applied at each time step.
+#
 #-----------------------------------------------------------------------
 #
-DO_SPP="FALSE"
-SPP_VAR_LIST=( "pbl" )
-SPP_MAG_LIST=( "0.2" ) #Variable "spp_prt_list" in input.nml
-SPP_LSCALE=( "150000.0" )
-SPP_TSCALE=( "21600.0" ) #Variable "spp_tau" in input.nml
-SPP_SIGTOP1=( "0.1" )
-SPP_SIGTOP2=( "0.025" )
-SPP_STDDEV_CUTOFF=( "1.5" )
+DO_SPP="false"
+SPP_VAR_LIST=( "pbl" "sfc" "mp" "rad" "gwd" )
+SPP_MAG_LIST=( "0.2" "0.2" "0.75" "0.2" "0.2" ) #Variable "spp_prt_list" in input.nml
+SPP_LSCALE=( "150000.0" "150000.0" "150000.0" "150000.0" "150000.0" )
+SPP_TSCALE=( "21600.0" "21600.0" "21600.0" "21600.0" "21600.0" ) #Variable "spp_tau" in input.nml
+SPP_SIGTOP1=( "0.1" "0.1" "0.1" "0.1" "0.1")
+SPP_SIGTOP2=( "0.025" "0.025" "0.025" "0.025" "0.025" )
+SPP_STDDEV_CUTOFF=( "1.5" "1.5" "2.5" "1.5" "1.5" )
+ISEED_SPP=( "4" "4" "4" "4" "4" )
+#
+#-----------------------------------------------------------------------
+#
+# Turn on SPP in Noah or RUC LSM (support for Noah MP is in progress).
+# Please be aware of the SDF that you choose if you wish to turn on LSM
+# SPP.
+#
+# SPP in LSM schemes is handled in the &nam_sfcperts namelist block 
+# instead of in &nam_sppperts, where all other SPP is implemented.
+#
+# The default perturbation frequency is determined by the fhcyc namelist 
+# entry.  Since that parameter is set to zero in the SRW App, use 
+# LSM_SPP_EACH_STEP to perturb every time step. 
+#
+# Perturbations to soil moisture content (SMC) are only applied at the 
+# first time step.
+#
+# LSM perturbations include SMC - soil moisture content (volume fraction),
+# VGF - vegetation fraction, ALB - albedo, SAL - salinity, 
+# EMI - emissivity, ZOL - surface roughness (cm), and STC - soil temperature.
+#
+# Only five perturbations at a time can be applied currently, but all seven
+# are shown below.  In addition, only one unique iseed value is allowed 
+# at the moment, and is used for each pattern.
+#
+DO_LSM_SPP="false" #If true, sets lndp_type=2
+LSM_SPP_TSCALE=( "21600" "21600" "21600" "21600" "21600" "21600" "21600" )
+LSM_SPP_LSCALE=( "150000" "150000" "150000" "150000" "150000" "150000" "150000" )
+ISEED_LSM_SPP=( "9" )
+LSM_SPP_VAR_LIST=( "smc" "vgf" "alb" "sal" "emi" "zol" "stc" )
+LSM_SPP_MAG_LIST=( "0.2" "0.001" "0.001" "0.001" "0.001" "0.001" "0.2" )
+LSM_SPP_EACH_STEP="true" #Sets lndp_each_step=.true.
 #
 #-----------------------------------------------------------------------
 # 
@@ -1936,3 +1973,6 @@ OMP_STACKSIZE_RUN_FCST="1024m"
 KMP_AFFINITY_RUN_POST="scatter"
 OMP_NUM_THREADS_RUN_POST="1"
 OMP_STACKSIZE_RUN_POST="1024m"
+#
+#-----------------------------------------------------------------------
+#

ush/generate_FV3LAM_wflow.sh

@@ -784,6 +784,12 @@ settings="\
     'do_shum': ${DO_SHUM},
     'do_sppt': ${DO_SPPT},
     'do_skeb': ${DO_SKEB},
+    'do_spp': ${DO_SPP},
+    'n_var_spp': ${N_VAR_SPP},
+    'n_var_lndp': ${N_VAR_LNDP},
+    'lndp_type': ${LNDP_TYPE},
+    'lndp_each_step': ${LSM_SPP_EACH_STEP},
+    'fhcyc': ${FHCYC_LSM_SPP_OR_NOT}, 
   }
 'nam_stochy': {
     'shum': ${SHUM_MAG},
@@ -855,7 +861,42 @@ done
 #
 settings="$settings
   }"
-
+#
+# Add the relevant SPP namelist variables to "settings" when running with 
+# SPP turned on.  Otherwise only include an empty "nam_sppperts" stanza.
+#
+settings="$settings
+'nam_sppperts': {"
+if [ "${DO_SPP}" = "TRUE" ]; then 
+    settings="$settings
+    'iseed_spp': [ $( printf "%s, " "${ISEED_SPP[@]}" ) ],
+    'spp_lscale': [ $( printf "%s, " "${SPP_LSCALE[@]}" ) ],
+    'spp_prt_list': [ $( printf "%s, " "${SPP_MAG_LIST[@]}" ) ],
+    'spp_sigtop1': [ $( printf "%s, " "${SPP_SIGTOP1[@]}" ) ],
+    'spp_sigtop2': [ $( printf "%s, " "${SPP_SIGTOP2[@]}" ) ],
+    'spp_stddev_cutoff': [ $( printf "%s, " "${SPP_STDDEV_CUTOFF[@]}" ) ],
+    'spp_tau': [ $( printf "%s, " "${SPP_TSCALE[@]}" ) ],
+    'spp_var_list': [ $( printf "%s, " "${SPP_VAR_LIST[@]}" ) ],"
+fi
+settings="$settings
+  }"
+#
+# Add the relevant LSM SPP namelist variables to "settings" when running with 
+# LSM SPP turned on.
+#
+settings="$settings
+'nam_sfcperts': {"
+if [ "${DO_LSM_SPP}" = "TRUE" ]; then 
+    settings="$settings
+    'lndp_type': ${LNDP_TYPE},
+    'lndp_tau': [ $( printf "%s, " "${LSM_SPP_TSCALE[@]}" ) ],
+    'lndp_lscale': [ $( printf "%s, " "${LSM_SPP_LSCALE[@]}" ) ],
+    'iseed_lndp': [ $( printf "%s, " "${ISEED_LSM_SPP[@]}" ) ],
+    'lndp_var_list': [ $( printf "%s, " "${LSM_SPP_VAR_LIST[@]}" ) ],
+    'lndp_prt_list': [ $( printf "%s, " "${LSM_SPP_MAG_LIST[@]}" ) ],"
+fi
+settings="$settings
+  }"
 print_info_msg $VERBOSE "
 The variable \"settings\" specifying values of the weather model's 
 namelist variables has been set as follows:

ush/machine/cheyenne.sh

@@ -59,13 +59,13 @@ RUN_CMD_UTILS='mpirun -np $nprocs'
 RUN_CMD_FCST='mpirun -np ${PE_MEMBER01}'
 RUN_CMD_POST='mpirun -np $nprocs'
 
-# MET Installation Locations
-MET_INSTALL_DIR="/glade/p/ral/jntp/MET/MET_releases/10.0.0"
-METPLUS_PATH="/glade/p/ral/jntp/MET/METplus/METplus-4.0.0"
-CCPA_OBS_DIR="/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ccpa/proc"
-MRMS_OBS_DIR="/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/mrms/proc"
-NDAS_OBS_DIR="/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ndas/proc"
-MET_BIN_EXEC="bin"
+# MET/METplus-Related Paths
+MET_INSTALL_DIR=${MET_INSTALL_DIR:-"/glade/p/ral/jntp/MET/MET_releases/10.0.0"}
+METPLUS_PATH=${METPLUS_PATH:-"/glade/p/ral/jntp/MET/METplus/METplus-4.0.0"}
+CCPA_OBS_DIR=${CCPA_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ccpa/proc"}
+MRMS_OBS_DIR=${MRMS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/mrms/proc"}
+NDAS_OBS_DIR=${NDAS_OBS_DIR:-"/glade/p/ral/jntp/UFS_SRW_app/develop/obs_data/ndas/proc"}
+MET_BIN_EXEC=${MET_BIN_EXEC:-"bin"}
 
 # Test Data Locations
 TEST_PREGEN_BASEDIR="/glade/p/ral/jntp/UFS_CAM/FV3LAM_pregen"

ush/machine/hera.sh

@@ -62,13 +62,13 @@ RUN_CMD_UTILS="srun"
 RUN_CMD_FCST="srun"
 RUN_CMD_POST="srun"
 
-# MET Installation Locations
-MET_INSTALL_DIR="/contrib/met/10.0.0"
-METPLUS_PATH="/contrib/METplus/METplus-4.0.0"
-CCPA_OBS_DIR="/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/ccpa/proc"
-MRMS_OBS_DIR="/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/mrms/proc"
-NDAS_OBS_DIR="/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/ndas/proc"
-MET_BIN_EXEC="bin"
+# MET/METplus-Related Paths
+MET_INSTALL_DIR=${MET_INSTALL_DIR:-"/contrib/met/10.0.0"}
+METPLUS_PATH=${METPLUS_PATH:-"/contrib/METplus/METplus-4.0.0"}
+CCPA_OBS_DIR=${CCPA_OBS_DIR:-"/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/ccpa/proc"}
+MRMS_OBS_DIR=${MRMS_OBS_DIR:-"/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/mrms/proc"}
+NDAS_OBS_DIR=${NDAS_OBS_DIR:-"/scratch2/BMC/det/UFS_SRW_app/develop/obs_data/ndas/proc"}
+MET_BIN_EXEC=${MET_BIN_EXEC:-"bin"}
 
 # Test Data Locations
 TEST_PREGEN_BASEDIR="/scratch2/BMC/det/FV3LAM_pregen"

ush/machine/wcoss_dell_p3.sh

@@ -68,13 +68,13 @@ RUN_CMD_UTILS="mpirun"
 RUN_CMD_FCST='mpirun -l -np ${PE_MEMBER01}'
 RUN_CMD_POST="mpirun"
 
-# MET Installation Locations
-MET_INSTALL_DIR="/gpfs/dell2/emc/verification/noscrub/emc.metplus/met/10.0.0"
-METPLUS_PATH="/gpfs/dell2/emc/verification/noscrub/emc.metplus/METplus/METplus-4.0.0"
-CCPA_OBS_DIR="/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/ccpa/proc"
-MRMS_OBS_DIR="/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/mrms/proc"
-NDAS_OBS_DIR="/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/ndas/proc"
-MET_BIN_EXEC="exec"
+# MET/METplus-Related Paths
+MET_INSTALL_DIR=${MET_INSTALL_DIR:-"/gpfs/dell2/emc/verification/noscrub/emc.metplus/met/10.0.0"}
+METPLUS_PATH=${METPLUS_PATH:-"/gpfs/dell2/emc/verification/noscrub/emc.metplus/METplus/METplus-4.0.0"}
+CCPA_OBS_DIR=${CCPA_OBS_DIR:-"/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/ccpa/proc"}
+MRMS_OBS_DIR=${MRMS_OBS_DIR:-"/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/mrms/proc"}
+NDAS_OBS_DIR=${NDAS_OBS_DIR:-"/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/obs_data/ndas/proc"}
+MET_BIN_EXEC=${MET_BIN_EXEC:-"exec"}
 
 # Test Data Locations
 TEST_PREGEN_BASEDIR="/gpfs/dell2/emc/modeling/noscrub/UFS_SRW_App/FV3LAM_pregen"