Update spack fork from spack develop 2022/12/22 #424

climbfuji · 2022-12-22T16:12:56Z

#WIP

Description

Updates to documentation, CI builds and package configurations (in config/common/packages.yaml and in container configs) to work with the updated spack code from JCSDA/spack#210. Important updates are:

Update py-scipy from 1.8.0 to 1.9.3 to fix build errors and duplicate packages
Let spack build Python for Ubuntu Linux CI builds
Let spack build Python in macOS instructions for local systems.

Note: For now, keep the Homebrew Python for macOS CI runs. We should change this (and the Linux instructions for local systems) to spack-built python in the near future. And consider doing the same on HPCs.

Update .gitmodules and submodule pointer for spack for code review and testing

Issues

Working towards #394 and #376.

Dependencies

waiting on Update jcsda_emc_spack_stack from spack develop 2022/12/22 spack#210

Testing

CI tests passed for 8ae66fb
Successfully installed on @climbfuji's macOS (skylab-dev template + jedi-tools-env)

…d testing

climbfuji · 2022-12-22T17:48:00Z

@AlexanderRichert-NOAA Do you understand this error?

/Users/runner/work/spack-stack/spack-stack/spack/var/spack/repos/builtin/packages/bacio/package.py:44, in patch:
         43    def patch(self):
  >>     44        if self.spec.satisifes("@@2.4.1"):
         45            filter_file(".*", "2.4.1", "VERSION")

The code doesn't have two @@, only one: https://github.com/NOAA-EMC/spack/pull/210/files#diff-8112aa20228dc1ee61fe9020c28d297f7d62931810c3c2c27a86cdf17abc87ac

AlexanderRichert-NOAA · 2022-12-22T17:56:23Z

@AlexanderRichert-NOAA Do you understand this error?
/Users/runner/work/spack-stack/spack-stack/spack/var/spack/repos/builtin/packages/bacio/package.py:44, in patch:
         43    def patch(self):
  >>     44        if self.spec.satisifes("@@2.4.1"):
         45            filter_file(".*", "2.4.1", "VERSION")
The code doesn't have two @@, only one: https://github.com/NOAA-EMC/spack/pull/210/files#diff-8112aa20228dc1ee61fe9020c28d297f7d62931810c3c2c27a86cdf17abc87ac

No. I pushed a fix based on the function name typo, but I'm not sure what's going on with the double "@" in the output there. I'm hoping it's just a bug in the error output...

climbfuji · 2022-12-22T20:55:46Z

@AlexanderRichert-NOAA Do you understand this error?
/Users/runner/work/spack-stack/spack-stack/spack/var/spack/repos/builtin/packages/bacio/package.py:44, in patch:
         43    def patch(self):
  >>     44        if self.spec.satisifes("@@2.4.1"):
         45            filter_file(".*", "2.4.1", "VERSION")
The code doesn't have two @@, only one: https://github.com/NOAA-EMC/spack/pull/210/files#diff-8112aa20228dc1ee61fe9020c28d297f7d62931810c3c2c27a86cdf17abc87ac
No. I pushed a fix based on the function name typo, but I'm not sure what's going on with the double "@" in the output there. I'm hoping it's just a bug in the error output...

If I had read the error more carefully ... self.spec.satisifes should be self.spec.satisfies. I'll push an update.

There are new CI failures for Linux, apparently because apparenty something requires a Python newer than 3.9 and the external we have on Linux CI (3.8). I'll check and maybe switch to spack-built python, wanted to do that a while ago already.

AlexanderRichert-NOAA · 2022-12-22T21:11:55Z

If I had read the error more carefully ... self.spec.satisifes should be self.spec.satisfies. I'll push an update.

There are new CI failures for Linux, apparently because apparenty something requires a Python newer than 3.9 and the external we have on Linux CI (3.8). I'll check and maybe switch to spack-built python, wanted to do that a while ago already.

The "satisifes" typo is the one I was referring to-- the fix has already been merged.

Let me know if you want me to look at the Python/CI matter. Which images are using 3.8?

climbfuji · 2022-12-22T21:13:50Z

If I had read the error more carefully ... self.spec.satisifes should be self.spec.satisfies. I'll push an update.
There are new CI failures for Linux, apparently because apparenty something requires a Python newer than 3.9 and the external we have on Linux CI (3.8). I'll check and maybe switch to spack-built python, wanted to do that a while ago already.

The "satisifes" typo is the one I was referring to-- the fix has already been merged.

Let me know if you want me to look at the Python/CI matter. Which images are using 3.8?

I'm already testing the Linux build with internal Python. It's a ubuntu20 image I think. The update of the CI system to use AWS codebuild with more powerful, faster and better controllable images is long planned, just waiting for the spack repository move to happen.

climbfuji · 2022-12-22T21:14:26Z

But, fwiw, I pulled the spack develop branch this morning, and it didn't have the bacio fix?

AlexanderRichert-NOAA · 2022-12-22T21:26:06Z

But, fwiw, I pulled the spack develop branch this morning, and it didn't have the bacio fix?

https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/bacio/package.py
spack/spack#34663

climbfuji · 2022-12-22T21:46:46Z

Guess I was 1hr too early !

…

On Dec 22, 2022, at 2:26 PM, Alex Richert ***@***.***> wrote: But, fwiw, I pulled the spack develop branch this morning, and it didn't have the bacio fix? https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/bacio/package.py <https://github.com/spack/spack/tree/develop/var/spack/repos/builtin/packages/bacio/package.py> spack/spack#34663 <spack/spack#34663> — Reply to this email directly, view it on GitHub <#424 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RJJ5HEH7FODSE7XEF3WOTBPTANCNFSM6AAAAAATG4KMYI>. You are receiving this because you authored the thread.

climbfuji · 2022-12-22T22:10:09Z

@AlexanderRichert-NOAA - I guess the new logic in bacio isn't working as expected. I'll go back to what we had in our spack fork until there is a proper fix in the authoritative repo?

    '/usr/local/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib;/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib64' '-DCMAKE_PREFIX_PATH:STRING=' '-DCMAKE_POSITION_INDEPENDENT_CODE:BOOL=ON' '/home/runner/work/spack-stack/spack-stack/cache/build_stage/spack-stage-bacio-2.4.1-pjld7qkpid7oxovqaxhbi7auxsip6gd5/spack-src'

1 error found in build log:
  >> 3    CMake Error at CMakeLists.txt:5 (project):
     4      VERSION "2.4.12.4.1" format invalid.
     5    
     6    
     7    -- Configuring incomplete, errors occurred!

AlexanderRichert-NOAA · 2022-12-22T22:23:09Z

@AlexanderRichert-NOAA - I guess the new logic in bacio isn't working as expected. I'll go back to what we had in our spack fork until there is a proper fix in the authoritative repo?

    '/usr/local/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib;/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib64' '-DCMAKE_PREFIX_PATH:STRING=' '-DCMAKE_POSITION_INDEPENDENT_CODE:BOOL=ON' '/home/runner/work/spack-stack/spack-stack/cache/build_stage/spack-stage-bacio-2.4.1-pjld7qkpid7oxovqaxhbi7auxsip6gd5/spack-src'

1 error found in build log:
  >> 3    CMake Error at CMakeLists.txt:5 (project):
     4      VERSION "2.4.12.4.1" format invalid.
     5    
     6    
     7    -- Configuring incomplete, errors occurred!

Oof. Looks like a Python bug. Depending on the Python version, the code

import re
print(re.sub(".*","2.4.1","2.4.1"))

gives different results. 3.6.8 gives "2.4.1", 3.8.12 and 3.10.5 give "2.4.12.4.1". Must be matching a beginning or end of line. I think the fix will be to change "*" to "+".

climbfuji · 2022-12-22T22:28:51Z

Let me try

…

On Dec 22, 2022, at 3:23 PM, Alex Richert ***@***.***> wrote: @AlexanderRichert-NOAA <https://github.com/AlexanderRichert-NOAA> - I guess the new logic in bacio isn't working as expected. I'll go back to what we had in our spack fork until there is a proper fix in the authoritative repo? '/usr/local/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib;/home/runner/work/spack-stack/spack-stack/envs/default/install/intel/2021.8.0/bacio-2.4.1-pjld7qk/lib64' '-DCMAKE_PREFIX_PATH:STRING=' '-DCMAKE_POSITION_INDEPENDENT_CODE:BOOL=ON' '/home/runner/work/spack-stack/spack-stack/cache/build_stage/spack-stage-bacio-2.4.1-pjld7qkpid7oxovqaxhbi7auxsip6gd5/spack-src' 1 error found in build log: >> 3 CMake Error at CMakeLists.txt:5 (project): 4 VERSION "2.4.12.4.1" format invalid. 5 6 7 -- Configuring incomplete, errors occurred! Oof. Looks like a Python bug. Depending on the Python version, the code import re print(re.sub(".*","2.4.1","2.4.1")) gives different results. 3.6.8 gives "2.4.1", 3.8.12 and 3.10.5 give "2.4.12.4.1". Must be matching a beginning or end of line. I think the fix will be to change "*" to "+". — Reply to this email directly, view it on GitHub <#424 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RPRXQNJ7ZXXYPPRWL3WOTIFPANCNFSM6AAAAAATG4KMYI>. You are receiving this because you authored the thread.

climbfuji · 2022-12-22T22:30:57Z

If that works, you can cherry-pick 36caf0a390557a49073097b40486b243e422b7e2 and update bacio in the authoritative repo. This way we won't have merge conflicts down the road.

…o feature/update_spack_from_authoritative_20221222

edwardhartnett · 2022-12-28T00:09:13Z

.gitmodules

+  ##branch = develop
+  #url = https://github.com/NOAA-EMC/spack
+  #branch = jcsda_emc_spack_stack
+  url = https://github.com/climbfuji/spack


Is this intentional? Seems like you are switching to your fork here?

That's needed for the CI tests to work properly. Until the spack PR is merged, the spack submodule pointer in spack-stack must point to the branch and fork to find the updated code. Procedure is: CI tests need to complete, then we merge the spack PR (pending approval, of course), then update the spack submodule pointer in spack-stack and revert .gitmodules. Then CI runs again, but can be bypassed if the code in the spack PR is identical with the final code in our (EMC) spack fork/branch - this is normally the case and can be assessed quickly (I do that after merging the spack PR).

BTW, the ufs-weather-model is using the same for its code commit procedures.

AlexanderRichert-NOAA

cool

…o feature/update_spack_from_authoritative_20221222

Update .gitmodules and submodule pointer for spack for code review an…

b7a8720

…d testing

Try to build Python with spack for Linux CI

a254444

Bug fix in .github/actions/setup-spack-stack/action.yaml

8cf5d75

Update submodule pointer for spack

14f26c6

climbfuji added 2 commits December 22, 2022 16:57

Merge branch 'develop' of https://github.com/noaa-emc/spack-stack int…

b8d5c43

…o feature/update_spack_from_authoritative_20221222

Update submodule pointer for spack

3421fe7

climbfuji force-pushed the feature/update_spack_from_authoritative_20221222 branch from 517ee99 to 3421fe7 Compare December 23, 2022 03:42

climbfuji added 3 commits December 23, 2022 06:10

Don't specify py-scipy and py-pythran versions

c211771

Update py-scipy version and unpin py-pythran

c36dd4f

Update for macOS instructions: build Python with spack

8ae66fb

climbfuji mentioned this pull request Dec 27, 2022

Update jcsda_emc_spack_stack from spack develop 2022/12/22 JCSDA/spack#210

Merged

climbfuji changed the title ~~DRAFT: Update spack fork from spack develop 2022/12/22~~ Update spack fork from spack develop 2022/12/22 Dec 27, 2022

climbfuji self-assigned this Dec 27, 2022

climbfuji marked this pull request as ready for review December 27, 2022 23:55

climbfuji requested review from srherbener, ulmononian, mathomp4 and AlexanderRichert-NOAA December 27, 2022 23:55

climbfuji requested a review from edwardhartnett December 27, 2022 23:56

edwardhartnett reviewed Dec 28, 2022

View reviewed changes

AlexanderRichert-NOAA approved these changes Dec 28, 2022

View reviewed changes

mathomp4 approved these changes Dec 28, 2022

View reviewed changes

climbfuji added 2 commits December 28, 2022 13:55

Merge branch 'develop' of https://github.com/noaa-emc/spack-stack int…

8548435

…o feature/update_spack_from_authoritative_20221222

Revert .gitmodules and update submodule pointer for spack

e2e80f7

climbfuji merged commit 19ff826 into JCSDA:develop Dec 28, 2022

climbfuji deleted the feature/update_spack_from_authoritative_20221222 branch December 28, 2022 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update spack fork from spack develop 2022/12/22 #424

Update spack fork from spack develop 2022/12/22 #424

climbfuji commented Dec 22, 2022 •

edited

Loading

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022 via email

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022 via email

climbfuji commented Dec 22, 2022

edwardhartnett Dec 28, 2022

climbfuji Dec 28, 2022 •

edited

Loading

climbfuji Dec 28, 2022

AlexanderRichert-NOAA left a comment

Update spack fork from spack develop 2022/12/22 #424

Update spack fork from spack develop 2022/12/22 #424

Conversation

climbfuji commented Dec 22, 2022 • edited Loading

Description

Issues

Dependencies

Testing

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022 via email

climbfuji commented Dec 22, 2022

AlexanderRichert-NOAA commented Dec 22, 2022

climbfuji commented Dec 22, 2022 via email

climbfuji commented Dec 22, 2022

edwardhartnett Dec 28, 2022

Choose a reason for hiding this comment

climbfuji Dec 28, 2022 • edited Loading

Choose a reason for hiding this comment

climbfuji Dec 28, 2022

Choose a reason for hiding this comment

AlexanderRichert-NOAA left a comment

Choose a reason for hiding this comment

climbfuji commented Dec 22, 2022 •

edited

Loading

climbfuji Dec 28, 2022 •

edited

Loading