Enable debugging features in CCPP physics caps, bug fixes and cleanup for CCPP variables and metadata, port model to XSEDE Expanse, switch WW3 from develop to dev/ufs-weather-model

[climbfuji]

PR Checklist

• Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• If new or updated input data is required by this PR, it is clearly stated in the text of the PR.

Description

This PR updates the submodule pointers for fv3atm (GFDL_atmos_cubed_sphere, ccpp-framework, ccpp-physics) and stochastic_physics for the changes described in detail in the associated PRs below:

It also ports the model to the XSede Expanse system (contributed by @MinsukJi-NOAA).

Switch WW3 from develop to dev/ufs-weather-model (i.e. pull #850 into this PR). This changes the results (metadata) for two regression tests, see #850 for more information. No change to input data.

Also: bug fix in run_compile.sh, replace UNIT_TEST with OPNREQ_TEST.

Update 10/22/2021: The bug fixes for issue #883 change the results of all regression tests using RUC LSM.

Issue(s) addressed

Fixes #883
Fixes NCAR/ccpp-framework#405

Testing

Regression tests were run on Cheyenne with Intel and GNU against the existing baselines on 10/04/2021. All tests passed.

RegressionTests_cheyenne.gnu.log
RegressionTests_cheyenne.intel.log

Update 10/22/2021: with the updates for RUC LSM and WW3, the following baselines change (confirmed by manually comparing the Hera Intel 20211019 and 20211022 baselines):

cpld_bmark_p7 (20130402.060000.out_pnt.points differs)
cpld_control_wave_p7 (20210322.180000.out_pnt.points differs)
fv3_gsd (RESTART/sfc_data.tile{1-6} differ, nothing else)
fv3_gsd_debug (RESTART/sfc_data.tile{1-6} differ, nothing else)
fv3_hrrr (RESTART/sfc_data.tile{1-6} differ, nothing else)
fv3_rap (RESTART/sfc_data.tile{1-6} differ, nothing else)

Full regression tests will be run on all tier-1 platforms when it is time to merge:

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3
• CI - 7f12415

Dependencies

• waiting on Metadata bug fixes, cleanup of active_gases_array use in RRTMGP, cleanup of Ferrier-Aligo variables, remove redundant consistency checks NCAR/ccpp-physics#749
• waiting on Rename --debug flag to --verbose, add new debugging capability to detect array size mismatches, pretty-print auto-generated caps NCAR/ccpp-framework#404
• waiting on Better error message from CCPP in case call to ccpp_physics_run for group 'fast_physics' failed NOAA-GFDL/GFDL_atmos_cubed_sphere#148
• waiting on https://github.com/NOAA-EMC/fv3atm/pull/407/files
• waiting on Update to stochastic_physics.F90: variable size of in/out surface perturbation namelist arrays NOAA-PSL/stochastic_physics#48
• waiting on Include Expanse platform in NEMS NOAA-EMC/NEMS#110

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: BL
Repo location: /glade/scratch/dtcufsrt/autort/tests/auto/pr/749328747/20211022103016/ufs-weather-model
Please manually delete: /glade/scratch/dtcufsrt/FV3_RT/rt_47577
Test control_thompson_no_aero_debug 023 failed failed
Test control_thompson_no_aero_debug 023 failed in run_test failed
Please make changes and add the following label back:
cheyenne-gnu-BL

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022163025/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_151428
Baseline creation and move successful
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022184926/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_149379
Test cpld_control_c192_p7 005 failed failed
Test cpld_control_c192_p7 005 failed in run_test failed
Please make changes and add the following label back:
jet-intel-BL

[climbfuji]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022163025/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_151428
Baseline creation and move successful
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022184926/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_149379
Test cpld_control_c192_p7 005 failed failed
Test cpld_control_c192_p7 005 failed in run_test failed
Please make changes and add the following label back:
jet-intel-BL

Sigh. The usual timeout. We need to do something about this test. I'll rerun the failed BL job manually, complete the baseline and add the RT label afterwards.

[DusanJovic-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022163025/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_151428
Baseline creation and move successful
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022184926/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_149379
Test cpld_control_c192_p7 005 failed failed
Test cpld_control_c192_p7 005 failed in run_test failed
Please make changes and add the following label back:
jet-intel-BL

Sigh. The usual timeout. We need to do something about this test. I'll rerun the failed BL job manually, complete the baseline and add the RT label afterwards.

If you want you can try adding changes I made in this branch:
https://github.com/DusanJovic-NOAA/ufs-weather-model/commits/ecf_tries

[climbfuji]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022163025/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_151428
Baseline creation and move successful
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/749328747/20211022184926/ufs-weather-model
Please manually delete: /lfs4/HFIP/h-nems/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_149379
Test cpld_control_c192_p7 005 failed failed
Test cpld_control_c192_p7 005 failed in run_test failed
Please make changes and add the following label back:
jet-intel-BL

Sigh. The usual timeout. We need to do something about this test. I'll rerun the failed BL job manually, complete the baseline and add the RT label afterwards.

If you want you can try adding changes I made in this branch: https://github.com/DusanJovic-NOAA/ufs-weather-model/commits/ecf_tries

Please remind me again before I work on my next PR in a bit more than a week. I am hesitant to update this PR so late in the process (most regression tests finished so far). Thanks!

[climbfuji]

@DusanJovic-NOAA All submodule PRs have been merged and I verified the hashes. Please review and merge if ok. Thanks!