New "atmanlfv3inc" Rocoto job (#2420)

This PR, a companion to GDASApp PR
[#983](NOAA-EMC/GDASApp#983), creates a new
Rocoto job called "atmanlfv3inc" that computes the FV3 atmosphere
increment from the JEDI variational increment using a JEDI OOPS app in
GDASApp, called fv3jedi_fv3inc.x, that replaces the GDASApp Python
script, jediinc2fv3.py, for the variational analysis. The "atmanlrun"
job is renamed "atmanlvar" to better reflect the role it plays of
running now one of two JEDI executables for the atmospheric analysis
jobs.

Previously, the JEDI variational executable would interpolate and write
its increment, during the atmanlrun job, to the Gaussian grid, and then
the python script, jediinc2fv3.py, would read it and then write the FV3
increment on the Gaussian grid during the atmanlfinal job. Following the
new changes, the JEDI increment will be written directly to the cubed
sphere. Then during the atmanlfv3inc job, the OOPS app will read it and
compute the FV3 increment directly on the cubed sphere and write it out
onto the Gaussian grid.

The reason for writing first to the cubed sphere grid is that otherwise
the OOPS app would have to interpolate twice, once from Gaussian to
cubed sphere before computing the increment and then back to the
Gaussian, since all the underlying computations in JEDI are done on the
native grid.

The motivation for this new app and job is that eventually we wish to
transition all intermediate data to the native cubed sphere grid, and
the OOPS framework allows us the flexibility to read and write to/from
any grid format we wish by just changing the YAML configuration file
rather than hardcoding. When we do switch to the cubed sphere, it will
be an easy transition. Moreover, it the computations the OOPS app will
be done with a compiled executable rather than an interpreted Python
script, providing some performance increase.

It has been tested with a cycling experiment with JEDI in both Hera and
Orion to show that it runs without issues, and I have compared the FV3
increments computed by the original and news codes. The delp and
hydrostatic delz increments, the key increments produced during this
step, differ by a relative error of 10^-7 and 10^-2 respectively. This
difference is most likely due to the original python script doing its
internal computation on the interpolated Gaussian grid, while the new
OOPS app does its computations on the native cubed sphere before
interpolating the the Gaussian grid.

env/AWSPW.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmanlfv3inc atmensanlrun aeroanlrun snowanl"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"

env/CONTAINER.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmanlfv3inc atmensanlrun aeroanlrun snowanl"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"

env/HERA.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmensanlrun aeroanlrun snowanl atmanlfv3inc"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"
@@ -56,13 +56,13 @@ elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}
     export wavempexec=${launcher}
     export wave_mpmd=${mpmd_opt}
 
-elif [[ "${step}" = "atmanlrun" ]]; then
+elif [[ "${step}" = "atmanlvar" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun} --cpus-per-task=${NTHREADS_ATMANL}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar} --cpus-per-task=${NTHREADS_ATMANLVAR}"
 
 elif [[ "${step}" = "atmensanlrun" ]]; then
 
@@ -82,6 +82,14 @@ elif [[ "${step}" = "aeroanlrun" ]]; then
     [[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max}
     export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun} --cpus-per-task=${NTHREADS_AEROANL}"
 
+elif [[ "${step}" = "atmanlfv3inc" ]]; then
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc} --cpus-per-task=${NTHREADS_ATMANLFV3INC}"
+
 elif [[ "${step}" = "snowanl" ]]; then
 
     nth_max=$((npe_node_max / npe_node_snowanl))

env/HERCULES.env

@@ -53,13 +53,21 @@ case ${step} in
     export wave_mpmd=${mpmd_opt}
 
  ;;
- "atmanlrun")
+ "atmanlvar")
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun} --cpus-per-task=${NTHREADS_ATMANL}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar} --cpus-per-task=${NTHREADS_ATMANLVAR}"
+ ;;
+ "atmanlfv3inc")
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc} --cpus-per-task=${NTHREADS_ATMANLFV3INC}"
  ;;
  "atmensanlrun")
 

env/JET.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmensanlrun aeroanlrun snowanl atmanlfv3inc"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"
@@ -44,13 +44,13 @@ elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}
     export wavempexec=${launcher}
     export wave_mpmd=${mpmd_opt}
 
-elif [[ "${step}" = "atmanlrun" ]]; then
+elif [[ "${step}" = "atmanlvar" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar}"
 
 elif [[ "${step}" = "atmensanlrun" ]]; then
 
@@ -80,6 +80,14 @@ elif [[ "${step}" = "snowanl" ]]; then
 
     export APRUN_APPLY_INCR="${launcher} -n 6"
 
+elif [[ "${step}" = "atmanlfv3inc" ]]; then
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc}"
+
 elif [[ "${step}" = "ocnanalbmat" ]]; then
 
     export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

env/ORION.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmensanlrun aeroanlrun snowanl atmanlfv3inc"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"
@@ -52,13 +52,13 @@ elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}
     export wavempexec=${launcher}
     export wave_mpmd=${mpmd_opt}
 
-elif [[ "${step}" = "atmanlrun" ]]; then
+elif [[ "${step}" = "atmanlvar" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun} --cpus-per-task=${NTHREADS_ATMANL}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar} --cpus-per-task=${NTHREADS_ATMANLVAR}"
 
 elif [[ "${step}" = "atmensanlrun" ]]; then
 
@@ -88,6 +88,14 @@ elif [[ "${step}" = "snowanl" ]]; then
 
     export APRUN_APPLY_INCR="${launcher} -n 6"
 
+elif [[ "${step}" = "atmanlfv3inc" ]]; then
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc} --cpus-per-task=${NTHREADS_ATMANLFV3INC}"
+
 elif [[ "${step}" = "ocnanalbmat" ]]; then
 
     export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"

env/S4.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmensanlrun aeroanlrun snowanl atmanlfv3inc"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen efcs epos"
     echo "postsnd awips gempak"
@@ -44,13 +44,13 @@ elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}
     export wavempexec=${launcher}
     export wave_mpmd=${mpmd_opt}
 
-elif [[ "${step}" = "atmanlrun" ]]; then
+elif [[ "${step}" = "atmanlvar" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar}"
 
 elif [[ "${step}" = "atmensanlrun" ]]; then
 
@@ -80,6 +80,14 @@ elif [[ "${step}" = "snowanl" ]]; then
 
     export APRUN_APPLY_INCR="${launcher} -n 6"
 
+elif [[ "${step}" = "atmanlfv3inc" ]]; then
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc}"
+
 elif [[ "${step}" = "ocnanalbmat" ]]; then
      echo "WARNING: ${step} is not enabled on S4!"
 

env/WCOSS2.env

@@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then
 
     echo "Must specify an input argument to set runtime environment variables!"
     echo "argument can be any one of the following:"
-    echo "atmanlrun atmensanlrun aeroanlrun snowanl"
+    echo "atmanlvar atmensanlrun aeroanlrun snowanl atmanlfv3inc"
     echo "anal sfcanl fcst post metp"
     echo "eobs eupd ecen esfc efcs epos"
     echo "postsnd awips gempak"
@@ -38,13 +38,13 @@ elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}
     export wavempexec="${launcher} -np"
     export wave_mpmd=${mpmd_opt}
 
-elif [[ "${step}" = "atmanlrun" ]]; then
+elif [[ "${step}" = "atmanlvar" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_atmanlrun))
+    nth_max=$((npe_node_max / npe_node_atmanlvar))
 
-    export NTHREADS_ATMANL=${nth_atmanlrun:-${nth_max}}
-    [[ ${NTHREADS_ATMANL} -gt ${nth_max} ]] && export NTHREADS_ATMANL=${nth_max}
-    export APRUN_ATMANL="${launcher} -n ${npe_atmanlrun}"
+    export NTHREADS_ATMANLVAR=${nth_atmanlvar:-${nth_max}}
+    [[ ${NTHREADS_ATMANLVAR} -gt ${nth_max} ]] && export NTHREADS_ATMANLVAR=${nth_max}
+    export APRUN_ATMANLVAR="${launcher} -n ${npe_atmanlvar}"
 
 elif [[ "${step}" = "atmensanlrun" ]]; then
 
@@ -74,6 +74,14 @@ elif [[ "${step}" = "snowanl" ]]; then
 
     export APRUN_APPLY_INCR="${launcher} -n 6"
 
+elif [[ "${step}" = "atmanlfv3inc" ]]; then
+
+    nth_max=$((npe_node_max / npe_node_atmanlfv3inc))
+
+    export NTHREADS_ATMANLFV3INC=${nth_atmanlfv3inc:-${nth_max}}
+    [[ ${NTHREADS_ATMANLFV3INC} -gt ${nth_max} ]] && export NTHREADS_ATMANLFV3INC=${nth_max}
+    export APRUN_ATMANLFV3INC="${launcher} -n ${npe_atmanlfv3inc}"
+
 elif [[ "${step}" = "anal" ]] || [[ "${step}" = "analcalc" ]]; then
 
     export OMP_PLACES=cores

jobs/JGLOBAL_ATM_ANALYSIS_FV3_INCREMENT

@@ -0,0 +1,37 @@
+#! /usr/bin/env bash
+
+source "${HOMEgfs}/ush/preamble.sh"
+export WIPE_DATA="NO"
+export DATA=${DATA:-${DATAROOT}/${RUN}atmanl_${cyc}}
+source "${HOMEgfs}/ush/jjob_header.sh" -e "atmanlfv3inc" -c "base atmanl atmanlfv3inc"
+
+##############################################
+# Set variables used in the script
+##############################################
+
+
+##############################################
+# Begin JOB SPECIFIC work
+##############################################
+
+
+###############################################################
+# Run relevant script
+
+EXSCRIPT=${GDASATMRUNSH:-${SCRgfs}/exglobal_atm_analysis_fv3_increment.py}
+${EXSCRIPT}
+status=$?
+[[ ${status} -ne 0 ]] && exit "${status}"
+
+##############################################
+# End JOB SPECIFIC work
+##############################################
+
+##############################################
+# Final processing
+##############################################
+if [[ -e "${pgmout}" ]] ; then
+  cat "${pgmout}"
+fi
+
+exit 0

jobs/JGLOBAL_ATM_ANALYSIS_RUN → jobs/JGLOBAL_ATM_ANALYSIS_VARIATIONAL

@@ -3,7 +3,7 @@
 source "${HOMEgfs}/ush/preamble.sh"
 export WIPE_DATA="NO"
 export DATA=${DATA:-${DATAROOT}/${RUN}atmanl_${cyc}}
-source "${HOMEgfs}/ush/jjob_header.sh" -e "atmanlrun" -c "base atmanl atmanlrun"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "atmanlvar" -c "base atmanl atmanlvar"
 
 ##############################################
 # Set variables used in the script
@@ -18,7 +18,7 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "atmanlrun" -c "base atmanl atmanlrun"
 ###############################################################
 # Run relevant script
 
-EXSCRIPT=${GDASATMRUNSH:-${SCRgfs}/exglobal_atm_analysis_run.py}
+EXSCRIPT=${GDASATMRUNSH:-${SCRgfs}/exglobal_atm_analysis_variational.py}
 ${EXSCRIPT}
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"

jobs/rocoto/atmanlfv3inc.sh

@@ -0,0 +1,24 @@
+#! /usr/bin/env bash
+
+source "${HOMEgfs}/ush/preamble.sh"
+
+###############################################################
+# Source UFSDA workflow modules
+. "${HOMEgfs}/ush/load_ufsda_modules.sh"
+status=$?
+[[ ${status} -ne 0 ]] && exit "${status}"
+
+export job="atmanlfv3inc"
+export jobid="${job}.$$"
+
+###############################################################
+# setup python path for workflow utilities and tasks
+wxflowPATH="${HOMEgfs}/ush/python:${HOMEgfs}/ush/python/wxflow/src"
+PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${wxflowPATH}"
+export PYTHONPATH
+
+###############################################################
+# Execute the JJOB
+"${HOMEgfs}/jobs/JGLOBAL_ATM_ANALYSIS_FV3_INCREMENT"
+status=$?
+exit "${status}"

jobs/rocoto/atmanlrun.sh → jobs/rocoto/atmanlvar.sh

@@ -8,7 +8,7 @@ source "${HOMEgfs}/ush/preamble.sh"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"
 
-export job="atmanlrun"
+export job="atmanlvar"
 export jobid="${job}.$$"
 
 ###############################################################
@@ -19,6 +19,6 @@ export PYTHONPATH
 
 ###############################################################
 # Execute the JJOB
-"${HOMEgfs}/jobs/JGLOBAL_ATM_ANALYSIS_RUN"
+"${HOMEgfs}/jobs/JGLOBAL_ATM_ANALYSIS_VARIATIONAL"
 status=$?
 exit "${status}"

parm/config/gfs/config.atmanlfv3inc

@@ -0,0 +1,14 @@
+#! /usr/bin/env bash
+
+########## config.atmanlfv3inc ##########
+# Atm Var Analysis specific
+
+echo "BEGIN: config.atmanlfv3inc"
+
+# Get task specific resources
+. "${EXPDIR}/config.resources" atmanlfv3inc
+
+export JEDIYAML=${PARMgfs}/gdas/atm/utils/fv3jedi_fv3inc_variational.yaml.j2
+export JEDIEXE=${EXECgfs}/fv3jedi_fv3inc.x
+
+echo "END: config.atmanlfv3inc"

parm/config/gfs/config.atmanlvar

@@ -0,0 +1,11 @@
+#! /usr/bin/env bash
+
+########## config.atmanlvar ##########
+# Atm Var Analysis specific
+
+echo "BEGIN: config.atmanlvar"
+
+# Get task specific resources
+. "${EXPDIR}/config.resources" atmanlvar
+
+echo "END: config.atmanlvar"