Building Albany using Spack

Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine. The following links provide some documentation on Spack:

• Spack web page https://spack.io
• Spack documentation is at http://spack.readthedocs.io/en/latest/
• Spack Google Group https://groups.google.com/forum/#!forum/spack
• Specific examples of ~/.spack/linux/compilers.yaml files are highlighted below
• Some slides by Xylar Asay-Davis (@xylar) about the Albany-supported spack build

A working Albany build is supported in the E3SM fork of the spack repo: https://github.com/E3SM-Project/spack.

Clone and setup Spack from E3SM-Project repo

	git clone git@github.com:E3SM-Project/spack.git
        cd spack
        git checkout develop
	. share/spack/setup-env.sh

This approach requires uploading the ssh-keys for your machine to github (under Settings). An alternative to the first line above is: https://github.com/E3SM-Project/spack.git if you prefer cloning using HTTPS instead of SSH, which does not require uploading ssh-keys.

Note that spack will try to download and install packages from the web. If your machine is behind a proxy, you will need to set proxies (export http_proxy=..., export https_proxy=...) prior to running . share/spack/setup-env.sh above.

Building and installing Albany using SPACK

	spack --insecure install --dirty --keep-stage albany

The --keep-stage option will ensure that the Albany build is kept after the installation of Albany is complete. This is required if you plan to run the Albany regression tests post-build.

If you are interested in building Albany for the purpose of installing MPAS-Albany on top of Albany, please use the following command:

            spack --insecure install --dirty --keep-stage albany+mpas

Note that Albany requires one to use one of the following compilers:

            Clang      4.0.0 or higher
            GCC        5.3.0 or higher
            Intel     17.0.0 or higher
            NVCC      9.2.88 or higher

Instructions are provided below for how to check what compiler you have, how to build a new compiler using spack, and how to tell spack with what compiler to build Albany.

Determining the installed location of Albany

	spack location -i albany

To cd to that location automatically, one can run the following command:

            spack cd albany

Running Albany tests

First, cd to the Albany build directory, which be default will be /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-build* (this can also be reached by running spack cd albany. From here, you can execute the command ctest -V to run the tests (or run the tests manually).

Accessing/recompiling Albany source code

If you have built Albany with the --keep-stage option, you can find the source code by default in the following directory: /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-src. If you wish to make a change to the source code and recompile the code, you simply make the chance, then cd to the build directory (by default /tmp/$USER/spack-stage/spack-stage-albany-develop-*/spack-build*) and type make -j 16. No module loading / environment specification should be necessary.

Misc spack installation notes

• Building some of the Albany dependencies might result in certificate errors. To ignore certificates, use the command

    spack --insecure install albany
  

  Note that ignoring ssh certificate issues can compromise your system's security. Note that spack --insecure install albany is equivalent to spack -k install albany. The --insecure option is in general required on machines behind firewalls (with proxies).

• To build Albany your current shell environment, instead of the SPACK environment

    spack install --dirty albany
  

• To build Albany using 16 processors with verbose build output

    spack install -j 16 -v albany
  

• To run the Albany regression tests post-build, change into the build stage directory and run them with CTest

    spack cd albany
    cd ../spack-build
    ctest
  

• Uninstalling a spackage: spack unistall albany

Compilers.yaml Examples

cee-compute Machines at Sandia

The following is an example compilers.yaml file. It is typically found in the following location on Linux: ~/.spack/$arch/compilers.yaml.

	compilers:
	- compiler:
	    environment: {}
	    extra_rpaths: []
	    flags: {}
	    modules: []
	    operating_system: rhel6
	    paths:
	      cc: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang
	      cxx: /sierra/sntools/SDK/compilers/clang/6.00-RHEL6/bin/clang++
	      f77: null
	      fc: null
	    spec: clang@6.0.0
	    target: x86_64
	- compiler:
	    environment:
	      set:
	        LD_LIBRARY_PATH : /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib:/sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/lib64
	    extra_rpaths: []
	    flags: {}
	    modules: []
	    operating_system: rhel6
	    paths:
	      cc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gcc
	      cxx: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/g++
	      f77: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
	      fc: /sierra/sntools/SDK/compilers/gcc/5.4.0-RHEL6/bin/gfortran
	    spec: gcc@5.4.0
	    target: x86_64

To generate a default compilers.yaml file, please run spack compiler find.

Changing the compiler used with spack

To see what compilers you have on your machine, you can run spack compilers. Lets say this command returns the following:

==> Available compilers
-- gcc rhel7-x86_64 ---------------------------------------------
gcc@7.2.0  gcc@6.1.0  gcc@4.8.5  gcc@4.4.7

and we wish to build Albany for MPAS with gcc-7.2.0. This can be done as follows:

                 spack --insecure install --dirty --keep-stage albany%gcc@7.2.0+mpas

Lets say the compiler you wish to use is not available on your system, e.g., it is gcc@8.4.0. One first needs to build the compiler using spack:

                 spack --insecure install --dirty --keep-stage gcc@8.4.0

Once that is done, one needs to make a note of the location in which the compiler was installed, and update one's compilers.yaml file to have an entry for the new compiler. Next, one can run spack compilers again and verify that gcc@8.4.0 shows up. Finally, to build Albany with gcc-8.4.0, one runs the following command:

                 spack --insecure install --dirty --keep-stage albany%gcc@7.2.0+mpas

If you wish to use a different pre-built compiler for the spack build, you can add an entry to it in the ~/.spack/$arch/compilers.yaml file, following the example above.

Changing the spack-stage directory

By default, spack will stage its builds in the /tmp/$USER/spack-stage directory. To change this, edit the spack/etc/spack/default/config.yaml, line 62 to specify the new path, e.g.,

                build_stage:
                   - /scratch/albany/nightlySpackBuild/spack-stage

Telling spack to use pre-built TPLs

In some cases, a user may not want spack to build every single TPL on which Albany depends; rather, one may want to have SPACK use pre-built TPLs. This can be accomplished by creating a ~/.spack/packages.yaml file that points to the pre-installed TPLs. This file will look as follows:

                packages:
                    openmpi:
                        paths:
                            openmpi@1.10.7: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/openmpi/1.10.7/gcc/10.1.0/base/7jgrwmo
                        buildable: False
                    cmake:
                        paths:
                            cmake@3.23.1: /projects/sems/install/rhel7-x86_64/sems/v2/utility/cmake/3.23.1/gcc/8.3.0/base/frlm6uo
                        buildable: False
                    boost:
                        paths:
                            boost@1.74.0: /projects/sems/install/rhel7-x86_64/sems/v2/tpl/boost/1.74.0/gcc/10.1.0/base/ua5ge5q
                        buildable: False