From 0dfc0d21d2e2f1ef3821517406a410677e917bda Mon Sep 17 00:00:00 2001 From: Alex Richert <82525672+AlexanderRichert-NOAA@users.noreply.github.com> Date: Tue, 9 Jan 2024 04:26:04 -0500 Subject: [PATCH] Update README.md to include Spack (#212) * Update README.md to include Spack * COMPONENTS: Update bufr version --- COMPONENTS | 2 +- README.md | 148 ++++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 107 insertions(+), 43 deletions(-) diff --git a/COMPONENTS b/COMPONENTS index 41cb751..c5f82ee 100644 --- a/COMPONENTS +++ b/COMPONENTS @@ -19,6 +19,6 @@ g2c | v1.6.2 | 1.6.2 crtm | v2.3.0 | 2.3.0 upp | upp_v10.0.0 | 10.0.0 wrf_io | v1.2.0 | 1.2.0 -bufr | bufr_v11.5.0 | 11.5.0 +bufr | v12.0.1 | 12.0.1 wgrib2 | v2.0.8-cmake-v6 | 2.0.8 -------------------------------------------------------- diff --git a/README.md b/README.md index 8427945..85c72e8 100644 --- a/README.md +++ b/README.md @@ -3,49 +3,68 @@ ## Introduction This is a collection of libraries commonly known as NCEPLIBS that are -required for several NCEP applications e.g. UFS, GSI, UPP, etc. For +required for several NCEP applications, e.g., UFS, GSI, UPP, etc. For general information about NCEPLIBS, the user is referred to the [Wiki](https://github.com/NOAA-EMC/NCEPLIBS/wiki). For full documentation of each package see https://noaa-emc.github.io/NCEPLIBS/. -For installing NCEPLIBS on NOAA HPC systems, see the -[hpc-stack](https://github.com/NOAA-EMC/hpc-stack) project. To install -the third-party libraries needed by NCEPLIBS, see -[NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external). +For installing NCEPLIBS on [NOAA HPC +systems](https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html), +see the +[documentation](https://spack-stack.readthedocs.io/en/latest/Overview.html#getting-started) +for the [spack-stack](https://github.com/JCSDA/spack-stack) project. The +documentation provides instructions for using spack-stack to [build software +stacks from +scratch](https://spack-stack.readthedocs.io/en/latest/CreatingEnvironments.html#creatingenvironment), +as well as to [build stacks that are chained to existing +environments](https://spack-stack.readthedocs.io/en/latest/AddingTestPackages.html#add-test-packages) +in order to re-use dependencies already provided by official spack-stack +deployments. **Note** that users are recommended to use existing spack-stack +installations as much as possible when using NOAA R&D systems, and to use +spack-stack as much as possible when developing for applications that +spack-stack is intended to support, namely, UFS applications, Global Workflow, +and JEDI applications (i.e., users developing the UFS Weather Model on a +personal machine are recommended to use their own installation of spack-stack). +Outside of these platforms and applications, users are recommended to install +NCEPLIBS either through CMake or through regular Spack (not spack-stack); see +instructions below. NCEPLIBS is comprised of the following libraries: - NCEP library | Description -----------------------|--------------------------------------------------- -[NCEPLIBS-bacio](https://github.com/NOAA-EMC/NCEPLIBS-bacio) | binary I/O for the NCEP models -[NCEPLIBS-bufr](https://github.com/NOAA-EMC/NCEPLIBS-bufr) | working with the BUFR format -[NCEPLIBS-g2](https://github.com/NOAA-EMC/NCEPLIBS-g2) | coding/decoding GRIB2 messages -[NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c) | C decoder/encoder routines for GRIB edition 2 -[NCEPLIBS-g2tmpl](https://github.com/NOAA-EMC/NCEPLIBS-g2tmpl) | Utilities for GRIB2 templates -[NCEPLIBS-gfsio](https://github.com/NOAA-EMC/NCEPLIBS-gfsio) | convert GFS Gaussian output into grib output -[NCEPLIBS-ip](https://github.com/NOAA-EMC/NCEPLIBS-ip) | general interpolation library -[NCEPLIBS-landsfcutil](https://github.com/NOAA-EMC/NCEPLIBS-landsfcutil) | initializing land-surface states -[NCEPLIBS-ncio](https://github.com/NOAA-EMC/NCEPLIBS-ncio) | read the GFS forecast files -[NCEPLIBS-nemsio](https://github.com/NOAA-EMC/NCEPLIBS-nemsio) | I/O for the NCEP models using NEMS -[NCEPLIBS-nemsiogfs](https://github.com/NOAA-EMC/NCEPLIBS-nemsiogfs) | I/O for the NEMS-GFS model -[NCEPLIBS-sfcio](https://github.com/NOAA-EMC/NCEPLIBS-sfcio) | surface files I/O -[NCEPLIBS-sigio](https://github.com/NOAA-EMC/NCEPLIBS-sigio) | I/O on the sigma restart file of the global spectral model -[NCEPLIBS-sp](https://github.com/NOAA-EMC/NCEPLIBS-sp) | spectral transform library -[NCEPLIBS-w3emc](https://github.com/NOAA-EMC/NCEPLIBS-w3emc) | decoder/encoder routines for GRIB edition 1 -[NCEPLIBS-w3nco](https://github.com/NOAA-EMC/NCEPLIBS-w3nco) | decoder/encoder routines for GRIB edition 1 -[NCEPLIBS-wrf_io](https://github.com/NOAA-EMC/NCEPLIBS-wrf_io) | lightweight WRF-IO API library for Unified Post Processor (UPP) -[NCEPLIBS-ip2](https://github.com/NOAA-EMC/NCEPLIBS-ip2) | general interpolation library 2 + NCEP library | Spack package name | Description +----------------------|--------------------|----------------------------------------------------- +[NCEPLIBS-bacio](https://github.com/NOAA-EMC/NCEPLIBS-bacio) | bacio | Binary I/O for the NCEP models +[NCEPLIBS-bufr](https://github.com/NOAA-EMC/NCEPLIBS-bufr) | bufr | Working with the BUFR format +[NCEPLIBS-g2](https://github.com/NOAA-EMC/NCEPLIBS-g2) | g2 | Coding/decoding GRIB2 messages +[NCEPLIBS-g2c](https://github.com/NOAA-EMC/NCEPLIBS-g2c) | g2c | C decoder/encoder routines for GRIB edition 2 +[NCEPLIBS-g2tmpl](https://github.com/NOAA-EMC/NCEPLIBS-g2tmpl) | g2tmpl | Utilities for GRIB2 templates +[NCEPLIBS-gfsio](https://github.com/NOAA-EMC/NCEPLIBS-gfsio) | gfsio | Convert GFS Gaussian output into grib output +[NCEPLIBS-ip](https://github.com/NOAA-EMC/NCEPLIBS-ip) | ip | General interpolation library +[NCEPLIBS-landsfcutil](https://github.com/NOAA-EMC/NCEPLIBS-landsfcutil) | landsfcutil | Initializing land-surface states +[NCEPLIBS-ncio](https://github.com/NOAA-EMC/NCEPLIBS-ncio) | ncio | Read the GFS forecast files +[NCEPLIBS-nemsio](https://github.com/NOAA-EMC/NCEPLIBS-nemsio) | nemsio | I/O for the NCEP models using NEMS +[NCEPLIBS-nemsiogfs](https://github.com/NOAA-EMC/NCEPLIBS-nemsiogfs) | nemsiogfs | I/O for the NEMS-GFS model +[NCEPLIBS-sfcio](https://github.com/NOAA-EMC/NCEPLIBS-sfcio) | sfcio | Surface files I/O +[NCEPLIBS-sigio](https://github.com/NOAA-EMC/NCEPLIBS-sigio) | sigio | I/O on the sigma restart file of the global spectral model +[NCEPLIBS-sp](https://github.com/NOAA-EMC/NCEPLIBS-sp) | sp | Spectral transform library +[NCEPLIBS-w3emc](https://github.com/NOAA-EMC/NCEPLIBS-w3emc) | w3emc | Decoder/encoder routines for GRIB edition 1 +[NCEPLIBS-wrf_io](https://github.com/NOAA-EMC/NCEPLIBS-wrf_io) | wrf-io | Lightweight WRF-IO API library for Unified Post Processor (UPP) + +The [NCEPLIBS-w3nco](https://github.com/NOAA-EMC/NCEPLIBS-w3nco) and +[NCEPLIBS-ip2](https://github.com/NOAA-EMC/NCEPLIBS-ip2) libraries +('w3nco' and 'ip2' in Spack, respectively) are deprecated, and +NCEPLIBS-w3emc and NCEPLIBS-ip, respectively, should be used instead. The following projects are also related to the NCEPLIBS project: - - [NCEPLIBS-grib_util](https://github.com/NOAA-EMC/NCEPLIBS-grib_util) - - [NCEPLIBS-prod_util](https://github.com/NOAA-EMC/NCEPLIBS-prod_util) - - [UFS_UTILS](https://github.com/NOAA-EMC/UFS_UTILS) + - [NCEPLIBS-grib_util](https://github.com/NOAA-EMC/NCEPLIBS-grib_util) ('grib-util' in Spack) + - [NCEPLIBS-prod_util](https://github.com/NOAA-EMC/NCEPLIBS-prod_util) ('prod-util' in Spack) + - [UFS_UTILS](https://github.com/NOAA-EMC/UFS_UTILS) ('ufs-utils' in Spack) ## Required Software -A supported C/C++ and Fortran compiler (see table below). Other + - A supported C/C++ and Fortran compiler (see table below). Other versions may work, in particular if close to the versions listed below. @@ -54,7 +73,7 @@ below. | Intel | 18.0.3.222, 18.0.5.274, 19.0.2.187, 19.0.5.281, 19.1.0.166 | | GNU | 8.3.0, 9.X.0 | -A supported MPI library (see table below). Other versions may work, + - A supported MPI library (see table below). Other versions may work, in particular if close to the versions listed below. | MPI library | Supported (tested) versions | @@ -65,8 +84,8 @@ in particular if close to the versions listed below. | Intel MPI | 2018.0.4, 2019.6.154, 2020.0.166 | | SGI MPT | 2.19 | -A software stack consisting of the following third-party libraries -(TPL), compiled with the same compiler and MPI library (where + - A software stack consisting of the following third-party libraries, +compiled with the same compiler and MPI library (where applicable). Other versions may work, in particular if close to the versions listed below. To install the third-party libraries needed by NCEPLIBS, see @@ -83,18 +102,59 @@ NCEPLIBS, see | NetCDF | netcdf-c-4.7.3, netcdf-fortran-4.5.2 | The [NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external) -project provides an convenient way to install the required TPL on -supported platforms such as generic Linux/macOS systems. +project provides an convenient way to install the required third-party libraries +on supported platforms such as generic Linux/macOS systems. + +## Obtaining, Building, and Using NCEPLIBS + +### With Spack +All of the NCEPLIBS can be installed through the [Spack](https://spack.io/) +package manager. One significant benefit of using Spack is that it will +automatically install any dependencies needed, including CMake, or even +compilers. See the [Spack +documentation](https://spack.readthedocs.io/en/latest/basic_usage.html) for +details of configuring and executing Spack installations. Spack also has +extensive capabilities including source and build caching, parallel +installation, dynamically generating Tcl and Lmod environment modules, +and creating Docker and Singularity container images. + +In the simplest case, with the `spack` command loaded into the current +environment, run, for example, `spack install ip` to install the latest release +of the NCEPLIBS-ip library along with any needed dependencies. See the above +table for each package's name in Spack. The develop branch for each library can +be installed by specifying the 'develop' version, i.e., `spack install +w3emc@develop`. + +Here is a typical set of steps for installing and using NCEPLIBS using Spack, +assuming that Spack's +[prerequisites](https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites) +are already installed: +``` +git clone https://github.com/spack/spack +. spack/share/spack/setup-env.sh # assuming bash, zsh, or dash shell +spack external find cmake gmake # find external packages to save time +spack install ip@develop precision=4 ^sp@2.4.0 precision=4 +spack load ip +``` +The `spack load` command configures the $CMAKE_PREFIX_PATH variable, +in this example making the ip library available to downstream CMake builds. + +For any bugs or other issues with Spack-based installations, such as incorrect +dependencies in a package's recipe, please file an issue under the appropriate +NCEPLIBS repository. -## Obtaining, Building, Using NCEPLIBS +### With CMake +#### External dependencies +See [NCEPLIBS-external](https://github.com/NOAA-EMC/NCEPLIBS-external) for +building external dependencies. -### Obtaining the code +#### Obtaining the code Clone the repository: `git clone https://github.com/NOAA-EMC/NCEPLIBS` -### Building the libraries +#### Building the libraries `CMake` employs an out-of-source build. Create a directory for configuring the build and cd into it: @@ -102,14 +162,14 @@ configuring the build and cd into it: `mkdir -p build && cd build` Set the compilers, if needed, to match those being used for compiling -the TPL listed above: `FC`, `CC`, and `CXX` environment variables can +the third-party libraries listed above: `FC`, `CC`, and `CXX` environment variables can be used to point to the desired Fortran, C, and C++ compilers. Execute `cmake` from inside your build directory. `cmake -DCMAKE_INSTALL_PREFIX= ../NCEPLIBS` -If the TPL are not located in a path recognized by `cmake` +If the third-party libraries are not located in a path recognized by `cmake` e.g. `/usr/local`, it may be necessary to provide the appropriate environment variables e.g. `` or `CMAKE_PREFIX_PATH` so that `cmake` is able to locate these dependencies. @@ -158,11 +218,15 @@ the tarball, and use the cmake command line option `USE_LOCAL` as `-DUSE_LOCAL=ON`. This will use the source code from the download directory. -### Usage +## Usage -`NCEPLIBS` can be used in any application that uses `cmake` to +NCEPLIBS can be used in any application that uses `cmake` to configure and build by adding `-DCMAKE_PREFIX_PATH=` -to the cmake command line during configuration. +to the cmake command line during configuration. Note that if NCEPLIBS +are installed through Spack, loading the packages, either through +Spack-built environment modules or using the `spack load` command, +will populate the `$CMAKE_PREFIX_PATH` environment variable with +the appropriate paths. ## Disclaimer