failing spack build of nest@3.6 (current "ebrains version") without mpi

[muffgaga]

Describe the bug
When trying to build the ebrains environment (spack build recipe from the ebrains repository → @terhorstd :D) including nest@3.6 with MPI disabled for nest (i.e. nest@3.6~mpi, the variant was set via a packages.yaml setting; EBRAINS also activates python, openmp, optimize, gsl, shared, sonata, boost).
@terhorstd suggested to put it here, instead of the EBRAINS Software Distribution issue tracker… however it's probably coupled to the nest spack build recipe from EBRAINS, and related to the other packages installed by the EBRAINS SW Dist.

To Reproduce

1. Steps to reproduce the behavior:
   😬 — I used the BrainScaleS container image build flow (https://github.com/electronicvisions/yashchiki), modified the base image to Debian bookworm (approx. here, but I added a new "style" container with a new Jenkinsfile, etc.), added the ebrains spack repository (master branch, from 2023-11-15), converted the environment definition to a BundlePackage definition, and installed this "ebrains" package. Compared to the "collab" build, I set the non-mpi nest variant via packages.yaml.
2. Used parameters
   spack says, it used those settings: (/opt/spack is the spack folder in this particular container image build flow)

-DCMAKE_INSTALL_LIBDIR=lib -DSPACK_CXX_COMPILER=/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-12.2.0/gcc-10.3.0-ghvge6o5t3dyml6qtpyy2vifhyfipaa5/bin/g++ -Dwith-boost=ON -Dwith-mpi=OFF -Dwith-openmp=ON -Dwith-optimize=ON -Dwith-gsl=/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gsl-2.7.1-tcs5tb2w3ubm6k3hxjwov23b23nlqpp2 -Dwith-hdf5=ON -Dstatic-libraries=OFF

3. Command to run (technically I did spack install ebrains ^python@3.8.11 %gcc@10.3.0 (version constraints from EBRAINS env) but this probably doesn't help here)
   The command spack tries to execute:

==> [2023-11-14-11:19:12.474153] '/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/fasthome/mueller/code/spack_ebrains/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/nest-3.6-tfmwfahhxkclvme7wjkw2sknr5rjqirn' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DBUILD_TESTING:BOOL=OFF' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/fasthome/mueller/code/spack_ebrains/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/nest-3.6-tfmwfahhxkclvme7wjkw2sknr5rjqirn/lib;/fasthome/mueller/code/spack_ebrains/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/nest-3.6-tfmwfahhxkclvme7wjkw2sknr5rjqirn/lib64;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gsl-2.7.1-tcs5tb2w3ubm6k3hxjwov23b23nlqpp2/lib' '-DCMAKE_PREFIX_PATH:STRING=/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-pandas-2.0.1-ae2mx26ocflm77ntd22lku3kbhiu7xw6;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-h5py-3.8.0-pffn2rrage6lmbg6c3salg3ythtucs3q;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/hdf5-1.14.1-2-ter32bjrfi4k6uupjpl2zucriaelwkcs;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gsl-2.7.1-tcs5tb2w3ubm6k3hxjwov23b23nlqpp2;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/boost-1.79.0-qxgxikk734rk6gbecpu27xmebiufjo63;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-numpy-1.23.5-gewulkyfxfebaahalwmonaes2nyl5f4r;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-cython-0.29.33-mhkbgtcjnamrrp7sxke5fdietplrbcfr;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-setuptools-62.6.0-lre36zsjqpidgiptf7tq63673yshb5ds;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/libtool-2.4.7-yckfhnu5am6vbauswkmxjwqhoeifgxg3;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/python-3.8.11-hwbme7alsu2b727lph4wxm5d4kk3j37g;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gmake-4.4.1-ztj6rfwszd2wbmmyve533o3ukarbynyj;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/readline-8.2-esufs2es5yuqnfakgp2hq4hxdmsd3tui;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/ncurses-6.4-uqpqlqisz2gvdvao36rr4go7weubuitt;/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/pkg-config-0.29.2-nmt5eiwt7ikw2kycplbu5a6dnkcjz2fr' '-DCMAKE_INSTALL_LIBDIR=lib' '-DSPACK_CXX_COMPILER=/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-12.2.0/gcc-10.3.0-ghvge6o5t3dyml6qtpyy2vifhyfipaa5/bin/g++' '-Dwith-boost=ON' '-Dwith-mpi=OFF' '-Dwith-openmp=ON' '-Dwith-optimize=ON' '-Dwith-gsl=/opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gsl-2.7.1-tcs5tb2w3ubm6k3hxjwov23b23nlqpp2' '-Dwith-hdf5=ON' '-Dstatic-libraries=OFF' '/tmp/mueller/spack-stage/spack-stage-nest-3.6-tfmwfahhxkclvme7wjkw2sknr5rjqirn/spack-src'

4. How to see error
   spack install's exit code != 0

Expected behavior
Installation should complete successfully. (spack install exit code == 0)

Screenshots

…
-- Info: Target triple: x86_64-pc-linux 
-- Found Python: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/python-3.8.11-hwbme7alsu2b727lph4wxm5d4kk3j37g/bin/python3.8 (found suitable version "3.8.11", minimum required is "3.8") found components: Interpreter Development.Module 
-- Found Cython: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/py-cython-0.29.33-mhkbgtcjnamrrp7sxke5fdietplrbcfr/bin/cython (found suitable version "0.29.33", minimum required is "0.28.3") 
-- Found LTDL: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/libtool-2.4.7-yckfhnu5am6vbauswkmxjwqhoeifgxg3/lib/libltdl.so (found version "2.4.7") 
-- Found Readline: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/readline-8.2-esufs2es5yuqnfakgp2hq4hxdmsd3tui/lib/libreadline.so (found version "8.2") 
-- Found GSL: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/gsl-2.7.1-tcs5tb2w3ubm6k3hxjwov23b23nlqpp2/include (found version "2.7.1") 
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Found Boost: /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/boost-1.79.0-qxgxikk734rk6gbecpu27xmebiufjo63/lib/cmake/Boost-1.79.0/BoostConfig.cmake (found suitable version "1.79.0", minimum required is "1.69.0")  
-- Checking for module 'mpi-c'
--   No package 'mpi-c' found
-- Checking for module 'mpi-cxx'
--   No package 'mpi-cxx' found
CMake Error at /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND)
Call Stack (most recent call first):
  /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64/share/cmake-3.26/Modules/FindMPI.cmake:1837 (find_package_handle_standard_args)
  /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/hdf5-1.14.1-2-ter32bjrfi4k6uupjpl2zucriaelwkcs/cmake/hdf5-config.cmake:91 (find_package)
  /opt/spack/opt/spack/linux-debian12-sandybridge/gcc-10.3.0/cmake-3.26.3-oxzao6kvy5kcbayzcwzdfyktc4wlur64/share/cmake-3.26/Modules/FindHDF5.cmake:486 (find_package)
  cmake/ProcessOptions.cmake:608 (find_package)
  CMakeLists.txt:154 (nest_process_with_hdf5)


-- Configuring incomplete, errors occurred!

Desktop/Environment (please complete the following information):

• OS: Debian bookworm + packages that spack needs + spack installation of ebrains env
• BrowseR: none
• Shell: bash from Debian bookworm
• Python-Version: EBRAINS → Python 3.8.11
• NEST-Version: nest@3.6~mpi
• Installation: spack (EBRAINS package definition maintained by @terhorstd)

Additional context
This is all quite far from standard build flows as I am working on a EBRAINS "laptop" container image… i.e. something people can just download and get an software environment with the full EBRAINS Software Distribution installed.
@terhorstd This used cmake@3.26.3.

[muffgaga]

The spack build logs:

spack-configure-args.txt
spack-build-out.txt
spack-build-env.txt
spack-build-env-mods.txt
spack-build-01-cmake-out.txt

[muffgaga]

Eleni cross-checked for "official" EBRAINS collab builds that it also fails there when building without mpi: https://gitlab.ebrains.eu/technical-coordination/project-internal/devops/platform/ebrains-spack-builds/-/jobs/52502 (not-yet-public repo → @terhorstd again :))

[muffgaga]

Eleni added link dependency on hdf5 in the recipe and it seems to fix it…
i.e. depends_on('hdf5+cxx', when='@3.4.99:+sonata', type=('build', 'link', 'run')).

[muffgaga]

During testing of #3112, it was deduced that the hdf5 variant influences this behavior.
If hdf5 wasn't built with mpi enabled, the with/without options works flawlessly — i.e. the bug can only be triggered if hdf5 is built with mpi support, but nest without.

[muffgaga]

For the spack package I proposed to align nest and its (sonata-feature-optional) dependency on hdf, i.e. use hdf5 in an MPI version if NEST is to be built with MPI.

diff --git a/packages/nest/package.py b/packages/nest/package.py
index f88e8f3..ebd8484 100644
--- a/packages/nest/package.py
+++ b/packages/nest/package.py
@@ -93,7 +93,8 @@ class Nest(CMakePackage):
     depends_on('boost',             when="@2.16:+boost", type='build')
     depends_on('py-setuptools@:44.99.99',     when='@:2.15.99+python', type='build')
     depends_on('py-h5py',           when='@3.4.99:+sonata', type=('test', 'build', 'run'))
-    depends_on('hdf5+cxx',          when='@3.4.99:+sonata', type=('build', 'link', 'run'))
+    depends_on('hdf5+cxx+mpi',      when='@3.4.99:+sonata+mpi', type=('build', 'link', 'run'))
+    depends_on('hdf5+cxx~mpi',      when='@3.4.99:+sonata~mpi', type=('build', 'link', 'run'))
     depends_on('py-pandas',         when='@3.4.99:+sonata', type=('build', 'run'))
 
     depends_on('py-nose',           when='@:2.99.99+python+testsuite', type='test')

⇒ So technically, this avoids the issue for spack installs — it could be still reasonable to try within nest's cmake flow to detect this and fail with a "helpful" error message (e.g. if HDF5_ENABLE_PARALLEL is ON, but nest's ${with-mpi} isn't ⇒ error).