Port to Hercules

[DavidHuber-NOAA]

This ports EMC_verif-global to Hercules. Though there isn't a stack that has been tested for step 2/plotting jobs, this includes a first stab at support of those jobs.
Resolves #120

[DavidHuber-NOAA]

A global workflow experiment was run on Hercules from 2021110818 through 2021111900. Grid2Grid and Precip .stat files were generated for the period, but Grid2Obs failed to generate files. This will need to be investigated.

@malloryprow In the meantime, would you mind taking a look at the G2G and Precip files in /work/noaa/global/dhuber/archive/metplus_data/by_VSDB under the experiment name herc_160? Also, the log files can be found here: /work2/noaa/global/dhuber/para/com/herc_160/logs/202111*/gfsmetp*.log.

[malloryprow]

I'll take a look at this later this week. I need to do some things for EVS.

Does Hercules and Orion have a shared file system? I'm surprised the precip would have ran. I've never put any CCPA files over there. I'm assuming the grid2obs failed due to a similar reasons as I have never put any prepbufr files there. What is the rstprod policy on Hercules? Same as Orion?

[DavidHuber-NOAA]

@malloryprow Yes, you are correct about grid2obs -- missing prepbufr files. Yes, Hercules and Orion have a shared file system and they both allow for rstprod data.

I'm curious about the precip case as well -- I'll check that out a little closer.

[malloryprow]

The precip doesn't run the first cycle since it needs more files to build the 24 hour accumulation. So the job was successful that nothing ran like it was suppose to (but there will be no stat file).

[DavidHuber-NOAA]

It looks like the CCPA data is available in /work/noaa/ovp/mrow/obdata/ccpa_accum24hr.

[DavidHuber-NOAA]

And at least on first glance, the precip stat file look OK in /work/noaa/global/dhuber/archive/metplus_data/by_VSDB/precip/ccpa_accum24hr/00Z/herc_160.

[malloryprow]

Interesting! Hmmm okay I'll look more into this later this week. I'd love to see if we can get the standalone version working too, so that people can make plots.

[malloryprow]

RE: gfsmetpg2o not finding the prepbufr files...

It is looking for the rstprod files at /work/noaa/ovp/mrow/prepbufr/nam which is where the non-restricted files reside. There are a few places in ush/get_data_files.py that need to be updated to include Hercules like on line 1697.

[malloryprow]

Okay I committed those changes directly to the branch. Pull those in and we can see if the next gfsmetpg2o run goes okay.

[DavidHuber-NOAA]

I finally got back to a place where I could run the GW with EMC_verif-global enabled to try these changes out and they seem to have worked! Would you mind checking out the logs (/work2/noaa/global/dhuber/para/com/herc_final_160/logs/*/gfsmetp*) and the .stat products (/work/noaa/global/dhuber/archive/metplus_data/by_VSDB under the experiment name herc_final_160)?

[DavidHuber-NOAA]

@malloryprow Marking this PR ready for review.

[malloryprow]

It all looks good to me! I'll merge the PR, and then the new commit will need to be updated for the g-w, right?

[DavidHuber-NOAA]

Yep, I'll get it into PR NOAA-EMC/global-workflow#2239.