Skip to content

Switch to Rocky 9 built external packages on Hercules #2608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 31, 2024
Merged

Switch to Rocky 9 built external packages on Hercules #2608

merged 6 commits into from
May 31, 2024

Conversation

WalterKolczynski-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA commented May 20, 2024 •
edited
Loading

Description

The workflow was updated to use modules built on Rocky 9, but the external packages (like prepobs) were still pointing to the versions built on CentOS (Orion). This transitions to packages built on Rocky 9.

Updating of the tracker package has been deferred until later. As such, the tracker jobs have been disabled by returning immediately if they are on Hercules. Since these jobs are small, resource-wise, it should not meaningfully impact turnover time.

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? YES
    • Tracker has not yet been updated to build on Hercules
  • Does this change require a documentation update? NO

How has this been tested?

  • Cycled test on Hercules

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

@WalterKolczynski-NOAA WalterKolczynski-NOAA self-assigned this May 20, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 20, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 85d8dba to 1f1ac3e Compare May 20, 2024 05:10
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels May 20, 2024
@WalterKolczynski-NOAA
Copy link
Contributor Author

The only update needed to the packages was a minor change to the Fit2Obs spack-stack module path:

-prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core")
+prepend_path("MODULEPATH", "/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/gsi-addon-env/install/modulefiles/Core")

Changes are made in-place without changing the version number of the directory (yet).

@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels May 20, 2024
@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C48_ATM FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C48_ATM_1f1ac3e4/logs/2021032312/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels May 20, 2024
@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C48_S2SW FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_S2SW_1f1ac3e4

@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C48_ATM FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C48_ATM_1f1ac3e4

@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C96_atm3DVar FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfsgenesis.log
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/COMROOT/C96_atm3DVar_1f1ac3e4/logs/2021122100/gfstracker.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C96C48_hybatmDA FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96C48_hybatmDA_1f1ac3e4

@emcbot
Copy link

emcbot commented May 20, 2024

Experiment C96_atm3DVar FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2608/RUNTESTS/C96_atm3DVar_1f1ac3e4

@DavidHuber-NOAA
Copy link
Contributor

It looks like the TC_Tracker is not in the specified location:

hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

@WalterKolczynski-NOAA
Copy link
Contributor Author

It looks like the TC_Tracker is not in the specified location:

hercules-login-1[3] ~$ ls /work/noaa/global/glopara/git_rocky9/TC_tracker/
hercules-login-1[4] ~$

Yeah, this was a known thing. I thought I had tested this before I launched CI, but might've mixed it up with another test I did over the weekend.

@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label May 29, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA mentioned this pull request May 30, 2024
Closed
1 task
@WalterKolczynski-NOAA
Switch to Rocky 9 built external packages on Hercules
2a7211e 
The workflow was updated to use modules built on Rocky 9, but the
external packages (like prepobs) were still pointing to the versions
built on CentOS (Orion). This transitions to packages built on Rocky
9.
@WalterKolczynski-NOAA WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 1f1ac3e to 8904861 Compare May 30, 2024 01:40
@WalterKolczynski-NOAA
Disable tracker jobs on Hercules
ecac748 
The external tracker package is not yet available on Rocky 9, so a
hack is added to skip those jobs on Hercules until the package is
available. This will need to be extended to Orion if that machine
transitions to Rocky 9 before a package is available.

Refs NOAA-EMC#2639
@WalterKolczynski-NOAA WalterKolczynski-NOAA force-pushed the feature/migrate_external_rocky9 branch from 8904861 to ecac748 Compare May 30, 2024 01:42
@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 30, 2024
@emcbot emcbot removed the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label May 30, 2024
@emcbot
Copy link

emcbot commented May 30, 2024

Experiment C48_S2SW_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:27:19 PM

@emcbot
Copy link

emcbot commented May 30, 2024

Experiment C96C48_hybatmDA_d45b7ca5 SUCCESS on Wcoss2 at 05/30/24 06:42:22 PM

@WalterKolczynski-NOAA WalterKolczynski-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels May 30, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

Experiment C96_atm3DVar_extended_d45b7ca5 SUCCESS on Wcoss2 at 05/31/24 06:33:26 AM

@emcbot emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:15 PM
Experiment C48_S2SW_d45b7ca5 *** SUCCESS *** at 05/30/24 06:27:19 PM
Experiment C96C48_hybatmDA_d45b7ca5 *** SUCCESS *** at 05/30/24 06:42:22 PM
Experiment C96_atm3DVar_extended_d45b7ca5 *** SUCCESS *** at 05/31/24 06:33:26 AM

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2608

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 12aa1e9 into NOAA-EMC:develop May 31, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA deleted the feature/migrate_external_rocky9 branch May 31, 2024 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aerorahul aerorahul aerorahul approved these changes

@KateFriedman-NOAA KateFriedman-NOAA Awaiting requested review from KateFriedman-NOAA

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA

Assignees

@WalterKolczynski-NOAA WalterKolczynski-NOAA

Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@WalterKolczynski-NOAA @emcbot @DavidHuber-NOAA @aerorahul @TerrenceMcGuinness-NOAA