Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to contrib spack-stack on Jet #978

Conversation

InnocentSouopgui-NOAA
Copy link
Contributor

@InnocentSouopgui-NOAA InnocentSouopgui-NOAA commented Aug 28, 2024

DESCRIPTION OF CHANGES:

Problem: Following the failure of the storage lfs4 on Jet, spack stack moved to another storage space: Contrib. This makes all softwares relying on spack stack libraries on Jet to fail.

Solution: Update jet modulefiles to point to the new installation of spack stack.

TESTS CONDUCTED:

The changes affects only Jet as all modifications are localized in Jet modulefile.

  • Compile branch Jet using Intel
  • Extract Initial conditions for Global Workflow cycled experiment on Jet.
  • Global Workflow at resolution
    • C96/48
    • C192/96
    • C384/192
  • Run relevant consistency tests locally on Jet.

Optional test.
Cycle Global workflow at resolutions 96/48, 192/96 and 384/192

DEPENDENCIES:

None

DOCUMENTATION:

Not needed.

ISSUE:

If this PR is resolving or referencing one or more issues, in this repository or elsewhere, list them here. For example, "Fixes issue mentioned in #123" or "Related to bug in https://github.com/NOAA-EMC/other_repository/pull/63"

Fixes issue mentioned in #977
Refs NOAA-EMC/global-workflow#2841

@InnocentSouopgui-NOAA InnocentSouopgui-NOAA marked this pull request as ready for review September 3, 2024 18:43
@InnocentSouopgui-NOAA
Copy link
Contributor Author

@GeorgeGayno-NOAA Can you review this so that we move it forward? The PR on global workflow depends on this.

@GeorgeGayno-NOAA
Copy link
Collaborator

I think your branch in behind the 'develop' branch. Please merge in the latest updates.

@InnocentSouopgui-NOAA
Copy link
Contributor Author

I think your branch in behind the 'develop' branch. Please merge in the latest updates.

Done

@GeorgeGayno-NOAA
Copy link
Collaborator

To get the ice_blend consistency test to run, I made these changes to the driver script (load the grib-util module, then remove the definitions of COPYGB, COPYGB2, and CNVGRIB).

diff --git a/reg_tests/ice_blend/driver.jet.sh b/reg_tests/ice_blend/driver.jet.sh
index 721b6bb6..2746d759 100755
--- a/reg_tests/ice_blend/driver.jet.sh
+++ b/reg_tests/ice_blend/driver.jet.sh
@@ -32,6 +32,7 @@ source ../../sorc/machine-setup.sh > /dev/null 2>&1
 module use ../../modulefiles
 module load build.$target.intel
 module load wgrib2/2.0.8
+module load grib-util/1.3.0
 set +x
 module list
 set -x
@@ -51,9 +52,6 @@ if [ "$UPDATE_BASELINE" = "TRUE" ]; then
 fi

 export WGRIB=/apps/wgrib/1.8.1.0b/bin/wgrib
-export COPYGB=/lfs5/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/copygb
-export COPYGB2=/lfs5/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/copygb2
-export CNVGRIB=/mnt/lfs5/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/cnvgrib

@InnocentSouopgui-NOAA
Copy link
Contributor Author

InnocentSouopgui-NOAA commented Sep 6, 2024

@GeorgeGayno-NOAA thanks for catching that.
It should be /contrib/spack-stack/spack-stack-1.6.0 for the path to spack-stack. I completely missed that because I did not run it.

While we are at it, what are all the consistencies tests to run?
I caught everything that is used by global workflow and the making of ICS for global workflow.

Are there regression tests that can catch problems in most components of UFS_UTILS?

I need to do another round of checks/modification.

@GeorgeGayno-NOAA
Copy link
Collaborator

@GeorgeGayno-NOAA thanks for catching that. It should be /contrib/spack-stack/spack-stack-1.6.0 for the path to spack-stack. I completely missed that because I did not run it.

While we are at it, what are all the consistencies tests to run? I caught everything that is used by global workflow and the makin of ICS for global workflow.

Are there regression tests that can catch problems in most components of UFS_UTILS?

I need to do another round of checks/modification.

The consistency/regression tests are in ./reg_tests. Most tests have a 'jet' driver script. Other tests have a single script that works for all machines. For instructions on how to run, see each scripts prolog. I am running the tests now. I recommend you run them as well. Let me know if you have questions.

@InnocentSouopgui-NOAA
Copy link
Contributor Author

@GeorgeGayno-NOAA , when you get the chance, have another look at the changes.

@@ -47,7 +47,7 @@ pwd=$(pwd -P)
if [ $machine = "hera" ]; then
FIX_DIR="/scratch1/NCEPDEV/global/glopara/fix"
elif [ $machine = "jet" ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Jet directory does not exist.

@@ -36,7 +36,7 @@ if [ "$machine" = 'hera' ] ; then
elif [ "$machine" = 'wcoss2' ] ; then
FIX_ORO_INPUT=/lfs/h2/emc/global/noscrub/emc.global/FIX/fix/orog/20230615
elif [ "$machine" = 'jet' ] ; then
FIX_ORO_INPUT=/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix/orog/20230615
FIX_ORO_INPUT=/lfs5/HFIP/hfv3gfs/glopara/git/fv3gfs/fix/orog/20230615
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue here. This directory does not exist.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be /lfs5/HFIP/hfv3gfs/glopara/FIX/fix/orog/20230615.

@@ -54,7 +54,7 @@ fi

export HOMEufs=$PWD/../..

export HOMEreg=/lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube
export HOMEreg=/lfs5/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/chgres_cube
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This directory does not exist yet (I am creating them now). Did you run the consistency tests?

export MOM6_FIXDIR=/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix/mom6/20220805
BASELINE_ROOT=/lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/cpld_gridgen/baseline_data
STMP=${STMP:-/lfs5/HFIP/h-nems/$USER}
export MOM6_FIXDIR=/lfs5/HFIP/hfv3gfs/glopara/git/fv3gfs/fix/mom6/20220805
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MOM6_FIXDIR should be /lfs5/HFIP/hfv3gfs/glopara/FIX/fix/mom6/20220805

export COPYGB=/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/copygb
export COPYGB2=/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/copygb2
export CNVGRIB=/mnt/lfs4/HFIP/hfv3gfs/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-hrqavdi/bin/cnvgrib
export COPYGB=/contrib/spack-stack/spack-stack-1.6.0/envs/unified-env-rocky8/install/intel/2021.5.0/grib-util-1.3.0-74mdurc/bin/copygb
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove the export of COPYGB, COPYGB2 and CNVGRIB if you add module load grib-util/1.3.0 near the top of the script.

@@ -47,7 +47,7 @@ pwd=$(pwd -P)
if [ $machine = "hera" ]; then
FIX_DIR="/scratch1/NCEPDEV/global/glopara/fix"
elif [ $machine = "jet" ]; then
FIX_DIR="/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix"
FIX_DIR="/lfs5/HFIP/hfv3gfs/glopara/git_lfs4/fv3gfs/fix"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@KateFriedman-NOAA - I see multiple copies of the 'fixed' data on Jet. Is this the correct directory?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that is a copy from /lfs4. The official fix path on Jet is now: /lfs5/HFIP/hfv3gfs/glopara/FIX/fix

@GeorgeGayno-NOAA GeorgeGayno-NOAA merged commit 06eec5b into ufs-community:develop Sep 9, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants