Create separate GSI monitor jobs

[KateFriedman-NOAA]

What new functionality do you need?

The GSI monitor jobs (radmon, ozmon, minmon) currently run within the vrfy job. They need to be their own separate jobs. Break them out of the vrfy job as part of work for issue #235.

What are the requirements for the new functionality?

New, separate, toggle-able jobs in the rocoto mesh for each GSI monitor task.

Acceptance Criteria

Three new functional jobs.

TO DO

• Remove GSI monitor tasks from vrfy job
• Remove GSI monitor settings from config.vrfy
• Create rocoto job scripts for GSI monitor jobs
• Add GSI monitor jobs to setup scripts
• Update output location to COM and update JJOB scripts to use COM_* variables
• Add GSI monitor job output to archival

[KateFriedman-NOAA]

The current GFSv16 ecflow definition file shows the following triggers for the GSI monitoring jobs:

                task jgfs_atmos_vminmon
                  trigger ../analysis/jgfs_atmos_analysis == complete

                task jgdas_atmos_vminmon
                  trigger ../analysis/jgdas_atmos_analysis == complete
                task jgdas_atmos_verfrad
                  trigger ../analysis/jgdas_atmos_analysis_diag == complete
                task jgdas_atmos_verfozn
                  trigger ../analysis/jgdas_atmos_analysis_diag == complete

Will mimic these job dependencies in rocoto.

[KateFriedman-NOAA]

@EdwardSafford-NOAA I've run the GSI monitor jobs as separate rocoto tasks in a test on WCOSS2-Dogwood. They completed without error. I spot checked a few files and they reproduced between the test and control but can you take a look at the logs and output to confirm they did indeed run correctly? Thanks!

Test run

EXPDIR: /lfs/h2/emc/global/noscrub/kate.friedman/expdir/testmonitor
COMROT logs:
  /lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor/logs/2021122100/gdasverfozn.log
  /lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor/logs/2021122100/gdasverfrad.log
  /lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor/logs/2021122100/gdasvminmon.log
  /lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor/logs/2021122100/gfsvminmon.log
verfozn output: /lfs/h2/emc/global/noscrub/kate.friedman/monitor/oznmon/stats/testmonitor
verfrad output: /lfs/h2/emc/global/noscrub/kate.friedman/monitor/radmon/stats/testmonitor
vminmon output: /lfs/h2/emc/global/noscrub/kate.friedman/monitor/minmon/stats/testmonitor

Control run

I ran a control run using the develop branch. Output from the GSI monitor tasks (run within the vrfy job) can be found here on Dogwood:

EXPDIR: /lfs/h2/emc/global/noscrub/kate.friedman/expdir/ctrlmonitor
COMROT logs: /lfs/h2/emc/ptmp/kate.friedman/comrot/ctrlmonitor/logs/2021122100/g*svrfy.log
output: /lfs/h2/emc/global/noscrub/kate.friedman/monitor_ctrl

[EdwardSafford-NOAA]

@KateFriedman-NOAA The output and logs look good, though I do have a question. The location of the gsistat file is reported in the log file as /lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor/gfs.20211221/00//analysis/atmos/gfs.t00z.gsistat. I note the double // between hh and analysis. It's in both the test and control log files. It's not causing any problem on dogwood, but that may not be true on all platforms. The gsistat assignment is made in exgfs_atmos/vminmon.sh at about line 45 using $COM_ATMOS_ANALYSIS. Is that a known issue with $COM_ATMOS_ANALYSIS?

[WalterKolczynski-NOAA]

@EdwardSafford-NOAA The // is expected and harmless. The COM templates have a directory level for member to accommodate ensembles. For deterministic it is just left blank resulting in the //.

[KateFriedman-NOAA]

@KateFriedman-NOAA The output and logs look good

Thanks for checking and confirming @EdwardSafford-NOAA !