Re-enable the C48mx500_3DVarAOWCDA CI

[JessicaMeixner-NOAA]

What is wrong?

PR #2371 turned off the C48mx500_3DVarAOWCDA CI test shortly after it was re-enabled. This issue is to notify everyone that this test was turned off and track the necessary changes to re-enable this test.

What should have happened?

CI test should remain working.

What machines are impacted?

Hera

Steps to reproduce

Try running CI test, it doesn't work.

Additional information

FYI @guillaumevernieres @CatherineThomas-NOAA @aerorahul

Hopefully @aerorahul will provide additional details about what needs to be changed. Happy to help with this, but at this point I couldn't even get the test to set-up to get to where the issues are, which is either user-error or the issue is at the level of setting up the test.

Do you have a proposed solution?

No response

[JessicaMeixner-NOAA]

Okay there was some user error on my part. Now that I got past the first user-error issue, I think I got to the issue which is with
https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/config/gfs/config.prepoceanobs#L10-L11

I ran :

hfe03:/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/waves01/c48gdas04$ pslot=c48t06 RUNTESTS=/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/waves01/c48gdas04 ICSDIR_ROOT=/scratch1/NCEPDEV/global/glopara/data/ICSDIR SLURM_ACCOUNT=marine-cpu ./../global-workflow/workflow/create_experiment.py --yaml ../global-workflow/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml

And then got:

/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/waves01/c48gdas04/EXPDIR/c48t06/config.prepoceanobs: line 10: export: `}}/sorc/gdas.cd/parm/soca/obsprep/obsprep_config.yaml': not a valid identifier
/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/waves01/c48gdas04/EXPDIR/c48t06/config.prepoceanobs: line 11: export: `}}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml': not a valid identifier

which looks like:

export OBSPREP_YAML={{ HOMEgfs }}/sorc/gdas.cd/parm/soca/obsprep/obsprep_config.yaml
export OBS_LIST={{ HOMEgfs }}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml

Updating these two lines to change {{ HOMEgfs }} to ${HOMEgfs} resulted in a successful creation of experiment. I'll now try running it to see if there are other issues that are encountered.

[JessicaMeixner-NOAA]

I successfully am able to run the test with the following changes:

diff --git a/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml b/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml
index d9156e38..b972d3a4 100644
--- a/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml
+++ b/ci/cases/pr/C48mx500_3DVarAOWCDA.yaml
@@ -19,5 +19,4 @@ arguments:
 
 skip_ci_on_hosts:
   - orion
-  - hera
   - hercules
diff --git a/parm/config/gfs/yaml/defaults.yaml b/parm/config/gfs/yaml/defaults.yaml
index 9c90255c..4a069be3 100644
--- a/parm/config/gfs/yaml/defaults.yaml
+++ b/parm/config/gfs/yaml/defaults.yaml
@@ -32,12 +32,12 @@ snowanl:
 ocnanal:
   SOCA_INPUT_FIX_DIR: "/scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/72x35x25/soca"  # TODO: These need to go to glopara fix space.
   CASE_ANL: "C48"  # TODO: Check in gdasapp if used anywhere for SOCA
-  SOCA_OBS_LIST: "{{ HOMEgfs }}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml"  # TODO: This is also repeated in oceanprepobs
+  SOCA_OBS_LIST: "${HOMEgfs}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml"  # TODO: This is also repeated in oceanprepobs
   SOCA_NINNER: 100
   SABER_BLOCKS_YAML: ""
   NICAS_RESOL: 1
   NICAS_GRID_SIZE: 15000
 prepoceanobs:
-  SOCA_OBS_LIST: "{{ HOMEgfs }}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml"  # TODO: This is also repeated in ocnanal
-  OBSPREP_YAML: "{{ HOMEgfs }}/sorc/gdas.cd/parm/soca/obsprep/obsprep_config.yaml"
+  SOCA_OBS_LIST: "${HOMEgfs}/sorc/gdas.cd/parm/soca/obs/obs_list.yaml"  # TODO: This is also repeated in ocnanal
+  OBSPREP_YAML: "${HOMEgfs}/sorc/gdas.cd/parm/soca/obsprep/obsprep_config.yaml"
   DMPDIR: "/scratch1/NCEPDEV/global/glopara/data/experimental_obs"
diff --git a/sorc/ufs_model.fd b/sorc/ufs_model.fd
index 69886627..126db7a5 160000
--- a/sorc/ufs_model.fd
+++ b/sorc/ufs_model.fd
@@ -1 +1 @@
-Subproject commit 698866272846e8c0f8f61ddb1b20d6463460cd63

The update to the model hash to develop of a day or so ago is just because I forgot that I had updated that. I see no reason why it wouldn't work with the current ufs-weather-model hash. Should I go forward with a PR with these changes?

[aerorahul]

The {{ }} were part of eliminating ${ } in the yamls.

[aerorahul]

The {{ }} were part of eliminating ${ } in the yamls.

On a (more) careful look, these are string replacements into the config files, and so should retain the ${ }.
Apologies for the misinformed review early on.

[aerorahul]

closed via #2405