Create separate jobs for GFS MOS

[KateFriedman-NOAA]

What new functionality do you need?

Remove the GFS MOS task from the vrfy job and make it it's own separate job in the rocoto mesh. Subtask of issue #235.

What are the requirements for the new functionality?

New self-contained GFS MOS job.

Acceptance Criteria

New job exists and MOS is removed from vrfy job.

Suggest a solution (optional)

No response

[KateFriedman-NOAA]

gfs_mos family in production ecflow definition file:

      family gfs_mos
        edit QUEUESERV 'prod_transfer'
        edit PROJ 'MOS'
        family v5.4
          edit PACKAGEHOME '/lfs/h1/ops/%ENVIR%/packages/gfs_mos.%gfs_mos_ver%'
          family prep
            task jgfs_mos_stn_prep
              trigger /prod/primary/00/gfs/v16.3/gfs/atmos/post/jgfs_atmos_post_f096 == complete
              edit NTASK '3'
              edit PTILE '3'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_grd_prep
              trigger /prod/primary/00/gfs/v16.3/gfs/atmos/post/jgfs_atmos_post_f096 == complete
              edit NTASK '4'
              edit PTILE '4'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_ext_stn_prep
              trigger /prod/primary/00/gfs/v16.3/gfs/atmos/post == complete
              edit NTASK '2'
              edit PTILE '2'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_ext_grd_prep
              trigger /prod/primary/00/gfs/v16.3/gfs/atmos/post == complete
              edit NTASK '7'
              edit PTILE '7'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
          endfamily
          family forecast
            task jgfs_mos_stn_fcst
              trigger ../prep/jgfs_mos_stn_prep == complete
              edit NTASK '5'
              edit PTILE '5'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_grd_fcst
              trigger ../prep/jgfs_mos_stn_prep == complete and ../prep/jgfs_mos_grd_prep == complete
              edit NTASK '7'
              edit PTILE '7'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_ext_stn_fcst
              trigger ../prep/jgfs_mos_ext_stn_prep == complete and ../prdgen/jgfs_mos_stn_prdgen == complete
              edit NTASK '3'
              edit PTILE '3'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_ext_grd_fcst
              trigger ../prep/jgfs_mos_ext_stn_prep == complete and ../prep/jgfs_mos_ext_grd_prep == complete and jgfs_mos_grd_fcst == complete
              edit NTASK '7'
              edit PTILE '7'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
          endfamily
          family prdgen
            task jgfs_mos_stn_prdgen
              trigger ../forecast/jgfs_mos_stn_fcst == complete
              edit NTASK '1'
              edit PTILE '1'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_grd_prdgen
              trigger ../forecast/jgfs_mos_grd_fcst == complete and jgfs_mos_stn_prdgen == complete
              edit NTASK '18'
              edit PTILE '6'
              edit OMP_NUM_THREADS '16'
              edit NODES '3'
            task jgfs_mos_ext_stn_prdgen
              trigger ../forecast/jgfs_mos_ext_stn_fcst == complete and jgfs_mos_stn_prdgen == complete
              edit NTASK '1'
              edit PTILE '1'
              edit OMP_NUM_THREADS '1'
              edit NODES '1'
            task jgfs_mos_ext_grd_prdgen
              trigger ../forecast/jgfs_mos_ext_grd_fcst == complete and jgfs_mos_grd_prdgen == complete and jgfs_mos_ext_stn_prdgen == complete
              edit NTASK '17'
              edit PTILE '6'
              edit OMP_NUM_THREADS '16'
              edit NODES '3'
            task jgfs_mos_wx_prdgen
              trigger ../prdgen/jgfs_mos_grd_prdgen == complete
              edit NTASK '2'
              edit PTILE '2'
              edit OMP_NUM_THREADS '20'
              edit NODES '1'
            task jgfs_mos_wx_ext_prdgen
              trigger ../prdgen/jgfs_mos_ext_grd_prdgen == complete and jgfs_mos_wx_prdgen == complete
              edit NTASK '2'
              edit PTILE '2'
              edit OMP_NUM_THREADS '20'
              edit NODES '1'
          endfamily
        endfamily
      endfamily

Job logs in ops:

kate.friedman@clogin05:/lfs/h1/ops/prod/output/20231115> ls -ltr gfs*mos*_00* | grep -v atmos
-rw-r--r-- 1 ops.prod prod   876297 Nov 15 04:05 gfs_mos_grd_prep_00.o109866196
-rw-r--r-- 1 ops.prod prod  1369883 Nov 15 04:07 gfs_mos_stn_prep_00.o109866195
-rw-r--r-- 1 ops.prod prod   347760 Nov 15 04:10 gfs_mos_stn_fcst_00.o109866747
-rw-r--r-- 1 ops.prod prod   243234 Nov 15 04:10 gfs_mos_stn_prdgen_00.o109867380
-rw-r--r-- 1 ops.prod prod   369768 Nov 15 04:11 gfs_mos_grd_fcst_00.o109866748
-rw-r--r-- 1 ops.prod prod  1802496 Nov 15 04:38 gfs_mos_grd_prdgen_00.o109867457
-rw-r--r-- 1 ops.prod prod   268228 Nov 15 04:40 gfs_mos_wx_prdgen_00.o109872633
-rw-r--r-- 1 ops.prod prod  1030202 Nov 15 05:19 gfs_mos_ext_grd_prep_00.o109880354
-rw-r--r-- 1 ops.prod prod  1413465 Nov 15 05:22 gfs_mos_ext_stn_prep_00.o109880352
-rw-r--r-- 1 ops.prod prod   277345 Nov 15 05:25 gfs_mos_ext_stn_fcst_00.o109881093
-rw-r--r-- 1 ops.prod prod   251144 Nov 15 05:25 gfs_mos_ext_stn_prdgen_00.o109881753
-rw-r--r-- 1 ops.prod prod   366579 Nov 15 05:27 gfs_mos_ext_grd_fcst_00.o109881094
-rw-r--r-- 1 ops.prod prod  2586661 Nov 15 05:41 gfs_mos_ext_grd_prdgen_00.o109881979
-rw-r--r-- 1 ops.prod prod   268653 Nov 15 05:43 gfs_mos_wx_ext_prdgen_00.o109884682

[KateFriedman-NOAA]

Section of vrfy rocoto job in question:

###############################################################
echo
echo "=============== START TO RUN MOS ==============="
if [[ "${RUNMOS}" == "YES" && "${CDUMP}" == "gfs" ]]; then
    ${RUNGFSMOSSH} "${PDY}${cyc}"
fi

...and companion section of config.vrfy:

export RUNMOS="NO"                # whether to run entire MOS package

if [[ "${RUNMOS}" == "YES" && "${RUN}" == "gfs" ]]; then

    if [[ "${machine}" = "HERA" ]] ; then
        export RUNGFSMOSSH="${HOMEgfs}/scripts/run_gfsmos_master.sh.hera"
    else
        echo "WARNING: MOS package is not enabled on ${machine}!"
        export RUNMOS="NO"
        export RUNGFSMOSSH=""
    fi
fi

[KateFriedman-NOAA]

Walltimes from ecf scripts in ops package (/lfs/h1/ops/prod/packages/gfs_mos.v5.4.3/ecf):

jgfs_mos_ext_grd_fcst.ecf:#PBS -l walltime=0:10:00
jgfs_mos_ext_grd_prdgen.ecf:#PBS -l walltime=0:30:00
jgfs_mos_ext_grd_prep.ecf:#PBS -l walltime=0:10:00
jgfs_mos_ext_stn_fcst.ecf:#PBS -l walltime=0:20:00
jgfs_mos_ext_stn_prdgen.ecf:#PBS -l walltime=0:10:00
jgfs_mos_ext_stn_prep.ecf:#PBS -l walltime=0:15:00
jgfs_mos_grd_fcst.ecf:#PBS -l walltime=0:10:00
jgfs_mos_grd_prdgen.ecf:#PBS -l walltime=0:40:00
jgfs_mos_grd_prep.ecf:#PBS -l walltime=0:10:00
jgfs_mos_stn_fcst.ecf:#PBS -l walltime=0:10:00
jgfs_mos_stn_prdgen.ecf:#PBS -l walltime=0:10:00
jgfs_mos_stn_prep.ecf:#PBS -l walltime=0:10:00
jgfs_mos_wx_ext_prdgen.ecf:#PBS -l walltime=0:10:00
jgfs_mos_wx_prdgen.ecf:#PBS -l walltime=0:10:00

Resources from ecf scripts in ops package:

jgfs_mos_ext_grd_fcst.ecf:#PBS -l place=vscatter,select=1:ncpus=7:mpiprocs=7:ompthreads=1:mem=50GB
jgfs_mos_ext_grd_prdgen.ecf:#PBS -l place=vscatter,select=3:ncpus=96:mpiprocs=6:ompthreads=16:mem=30GB
jgfs_mos_ext_grd_prep.ecf:#PBS -l place=vscatter,select=1:ncpus=7:mpiprocs=7:ompthreads=1:mem=3GB
jgfs_mos_ext_stn_fcst.ecf:#PBS -l place=vscatter,select=1:ncpus=3:mpiprocs=3:ompthreads=1:mem=50GB:prepost=true
jgfs_mos_ext_stn_prdgen.ecf:#PBS -l place=vscatter,select=1:ncpus=1:mpiprocs=1:ompthreads=1:mem=15GB:prepost=true
jgfs_mos_ext_stn_prep.ecf:#PBS -l place=vscatter,select=1:ncpus=2:mpiprocs=2:ompthreads=1:mem=5GB
jgfs_mos_grd_fcst.ecf:#PBS -l place=vscatter,select=1:ncpus=7:mpiprocs=7:ompthreads=1:mem=50GB
jgfs_mos_grd_prdgen.ecf:#PBS -l place=vscatter,select=1:ncpus=72:mpiprocs=18:ompthreads=4:mem=20GB
jgfs_mos_grd_prep.ecf:#PBS -l place=vscatter,select=1:ncpus=4:mpiprocs=4:ompthreads=1:mem=16GB
jgfs_mos_stn_fcst.ecf:#PBS -l place=vscatter,select=1:ncpus=%NTASK%:mpiprocs=%PTILE%:ompthreads=1:mem=40GB
jgfs_mos_stn_prdgen.ecf:#PBS -l place=vscatter,select=1:ncpus=1:mpiprocs=1:ompthreads=1:mem=15GB:prepost=true
jgfs_mos_stn_prep.ecf:#PBS -l place=vscatter,select=1:ncpus=3:mpiprocs=3:ompthreads=1:mem=5GB
jgfs_mos_wx_ext_prdgen.ecf:#PBS -l place=vscatter,select=1:ncpus=4:mpiprocs=2:ompthreads=2:mem=10GB
jgfs_mos_wx_prdgen.ecf:#PBS -l place=vscatter,select=1:ncpus=4:mpiprocs=2:ompthreads=2:mem=10GB

Job script from ecf scripts in ops package:

jgfs_mos_ext_grd_fcst.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_GRD_FORECAST
jgfs_mos_ext_grd_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_GRD_PRDGEN
jgfs_mos_ext_grd_prep.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_GRD_PREP
jgfs_mos_ext_stn_fcst.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_STN_FORECAST
jgfs_mos_ext_stn_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_STN_PRDGEN
jgfs_mos_ext_stn_prep.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_EXT_STN_PREP
jgfs_mos_grd_fcst.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_GRD_FORECAST
jgfs_mos_grd_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_GRD_PRDGEN
jgfs_mos_grd_prep.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_GRD_PREP
jgfs_mos_stn_fcst.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_STN_FORECAST
jgfs_mos_stn_fcst.ecf_old:${HOMEgfs_mos}/jobs/JGFSMOS_STN_FORECAST
jgfs_mos_stn_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_STN_PRDGEN
jgfs_mos_stn_prep.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_STN_PREP
jgfs_mos_wx_ext_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_WX_EXT_PRDGEN
jgfs_mos_wx_prdgen.ecf:${HOMEgfs_mos}/jobs/JGFSMOS_WX_PRDGEN