Segfault in Spack build of trilinos-for-albany with intel

[xylar]

This is not a new issue but one I want to revisit. When I try to build trilinos-for-albany on Chrysalis with intel and OpenMPI, I see:

A long error message with a segfault

cd /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/belos/epetra/src && /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/gcc-9.3.0/cmake-3.19.1-yisciec/bin/cmake -E cmake_link_script CMakeFiles/belosepetra.dir/link.txt --verbose=1
/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/openmpi-4.1.3-pin4k7o/bin/mpic++ -fPIC -O2 -g -DNDEBUG -shared -Wl,-soname,libbelosepetra.so.13 -o libbelosepetra.so.13.5 CMakeFiles/belosepetra.dir/BelosEpetraAdapter.cpp.o CMakeFiles/belosepetra.dir/BelosEpetraOperator.cpp.o CMakeFiles/belosepetra.dir/BelosEpetraUtils.cpp.o CMakeFiles/belosepetra.dir/Belos_Details_Epetra_registerLinearSolverFactory.cpp.o CMakeFiles/belosepetra.dir/Belos_Details_Epetra_registerSolverFactory.cpp.o  -Wl,-rpath,/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/belos/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/xpetra/sup:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/xpetra/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/thyra/adapters/epetraext/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/epetraext/src:/gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/hdf5-1.10.7-eewgp6v/lib:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/thyra/adapters/tpetra/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/tpetra/core/ext:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/tpetra/core/inout:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/tpetra/core/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/tpetra/core/compat:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/tpetra/tsqr/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/thyra/adapters/epetra/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/thyra/core/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/rtop/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/aztecoo/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/triutils/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/epetra/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/numerics/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/remainder/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/kokkoscomm/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/comm/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/kokkoscompat/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/parameterlist/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/parser/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/teuchos/core/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/kokkos-kernels/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/kokkos/algorithms/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/kokkos/containers/src:/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/kokkos/core/src:/lcrc/soft/climate/compass/chrysalis/spack/spack_for_mache_1.8.0/opt/spack/linux-rhel8-zen2/intel-20.0.4/metis-5.1.0-fvpnjgznlef67rs2jblxnjoxjaue2iyj/lib: ../../src/libbelos.so.13.5 ../../../xpetra/sup/libxpetra-sup.so.13.5 ../../../xpetra/src/libxpetra.so.13.5 ../../../thyra/adapters/epetraext/src/libthyraepetraext.so.13.5 ../../../epetraext/src/libepetraext.so.13.5 /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/hdf5-1.10.7-eewgp6v/lib/libhdf5.so /usr/lib64/libz.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/hdf5-1.10.7-eewgp6v/lib/libhdf5_hl.so ../../../thyra/adapters/tpetra/src/libthyratpetra.so.13.5 ../../../tpetra/core/ext/libtpetraext.so.13.5 ../../../tpetra/core/inout/libtpetrainout.so.13.5 ../../../tpetra/core/src/libtpetra.so.13.5 ../../../tpetra/core/compat/libtpetraclassic.so.13.5 ../../../tpetra/tsqr/src/libkokkostsqr.so.13.5 ../../../thyra/adapters/epetra/src/libthyraepetra.so.13.5 ../../../thyra/core/src/libthyracore.so.13.5 ../../../rtop/src/librtop.so.13.5 ../../../aztecoo/src/libaztecoo.so.13.5 ../../../triutils/src/libtriutils.so.13.5 ../../../epetra/src/libepetra.so.13.5 ../../../teuchos/numerics/src/libteuchosnumerics.so.13.5 ../../../teuchos/remainder/src/libteuchosremainder.so.13.5 ../../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.13.5 ../../../teuchos/comm/src/libteuchoscomm.so.13.5 ../../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.13.5 ../../../teuchos/parameterlist/src/libteuchosparameterlist.so.13.5 ../../../teuchos/parser/src/libteuchosparser.so.13.5 ../../../teuchos/core/src/libteuchoscore.so.13.5 ../../../kokkos-kernels/src/libkokkoskernels.so.13.5 /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_intel_lp64.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_sequential.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_core.so /lib64/libpthread.so /lib64/libm.so /lib64/libdl.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_intel_lp64.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_sequential.so /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/intel-mkl-2020.4.304-g2qaxzf/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64/libmkl_core.so /lib64/libpthread.so /lib64/libm.so /lib64/libdl.so ../../../kokkos/algorithms/src/libkokkosalgorithms.so.13.5 ../../../kokkos/containers/src/libkokkoscontainers.so.13.5 ../../../kokkos/core/src/libkokkoscore.so.13.5 /usr/lib64/libdl.so /lcrc/soft/climate/compass/chrysalis/spack/spack_for_mache_1.8.0/opt/spack/linux-rhel8-zen2/intel-20.0.4/metis-5.1.0-fvpnjgznlef67rs2jblxnjoxjaue2iyj/lib/libmetis.so

          ": internal error: ** The compiler has encountered an unexpected problem.
** Segmentation violation signal raised. **
Access violation or stack overflow. Please contact Intel Support for assistance.

icpc: error #10105: /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/gcc-9.3.0/intel-20.0.4-kodw73g/compilers_and_libraries_2020.4.304/linux/bin/intel64/mcpcom: core dumped
icpc: warning #10102: unknown signal(1415383120)
icpc: error #10106: Fatal error in /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/gcc-9.3.0/intel-20.0.4-kodw73g/compilers_and_libraries_2020.4.304/linux/bin/intel64/mcpcom, terminated by unknown
icpc: error #10014: problem during multi-file optimization compilation (code 1)
make[2]: *** [packages/stk/stk_unit_test_utils/stk_unit_test_utils/stk_mesh_fixtures/CMakeFiles/stk_mesh_fixtures.dir/build.make:460: packages/stk/stk_unit_test_utils/stk_unit_test_utils/stk_mesh_fixtures/libstk_mesh_fixtures.so.13.5] Error 1
make[2]: Leaving directory '/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi'
make[1]: *** [CMakeFiles/Makefile2:14783: packages/stk/stk_unit_test_utils/stk_unit_test_utils/stk_mesh_fixtures/CMakeFiles/stk_mesh_fixtures.dir/all] Error 2
cd /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi/packages/belos/epetra/src && /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/gcc-9.3.0/cmake-3.19.1-yisciec/bin/cmake -E cmake_symlink_library libbelosepetra.so.13.5 libbelosepetra.so.13 libbelosepetra.so
make[2]: Leaving directory '/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi'
[ 80%] Built target belosepetra
make[1]: Leaving directory '/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-trilinos-for-albany-develop-we2yuzibwzgdeomqm5qiqy7ndkisjkpj/spack-build-we2yuzi'
make: *** [Makefile:174: all] Error 2

I believe this is the same issue I have seen previously with Intel on all machines I've tried.

It would be really great to get this resolved, since Intel and OpenMPI are the production compilers on Chrysalis for E3SM.

[xylar]

@ikalash, @mperego, @bartgol, @jewatkins

I'm just reviving this issue in the hope that we can resolve it in the not-too-distant future. It's not super urgent, just bothersome.

[xylar]

I see the same issue on Chrysalis when I use Intel and Intel-MPI (instead of OpenMPI).

Here is a gist with a shell script and yaml file that can be used to reproduce the issue:
https://gist.github.com/xylar/4fa8c2a7a54335068bedbb38a64d584c

[bartgol]

Ugh, ICEs are annoying, and often require seemingly ineffective changes (like split a cpp file in two, or remove a const), with lots of bisections iterations to identify the problematic code snippet.

That said, from the error msg, it appears it's trying to build the stk_unit_test_utils package, which I think we can do away with. Can you try to set Trilinos_ENABLE_STKUnit_test_utils:BOOL=OFF? This is always off in my trilinos builds, along with a few other stk sub-packages:

  -D Trilinos_ENABLE_STK:BOOL=ON                                  \
  -D Trilinos_ENABLE_STKDoc_tests:BOOL=OFF                        \
  -D Trilinos_ENABLE_STKIO:BOOL=ON                                \
  -D Trilinos_ENABLE_STKMesh:BOOL=ON                              \
  -D Trilinos_ENABLE_STKSearch:BOOK=ON                            \
  -D Trilinos_ENABLE_STKSearchUtil:BOOL=OFF                       \
  -D Trilinos_ENABLE_STKTopology:BOOL=ON                          \
  -D Trilinos_ENABLE_STKTransfer:BOOL=ON                          \
  -D Trilinos_ENABLE_STKUnit_tests:BOOL=OFF                       \
  -D Trilinos_ENABLE_STKUnit_test_utils:BOOL=OFF                  \
  -D Trilinos_ENABLE_STKUtil:BOOL=ON    

[jewatkins]

I see the same issue on Chrysalis when I use Intel and Intel-MPI (instead of OpenMPI).

Here is a gist with a shell script and yaml file that can be used to reproduce the issue: https://gist.github.com/xylar/4fa8c2a7a54335068bedbb38a64d584c

Can this be used to reproduce on any machine? I remember there was some issue with intel + spack a while back but that may have been resolved. Part of the issue is that everyone has different Trilinos configure scripts. It would be nice to have an up-to-date master one that we can all contribute to and have all our nightly testing conform to.

We are currently testing up to intel 19 and it looks like trilinos is doing the same? https://testing.sandia.gov/cdash/index.php?subproject=STK&project=Trilinos

We do have intel 20.2.254 on blake, we could try to update our testing there. I do recall seeing some issues with oneapi but I think they were mostly tpl issues.

[xylar]

@jewatkins, we should have a discussion about this. This recipe is specific to Chrysalis. Similar recipes are needed for other supported machines. As I presented on Tuesday, we have a python package mache that handles generating these build scripts and yaml files (as well as snippets that can be part of activation scripts) to handle different machines. But there's not an easy way to make this particular type of recipe machine independent and indeed the point from my perspective is to mimic E3SM, which is decidedly not machine independent.

[xylar]

I can make similar recipes for Compy, Cori or Anvil. We don't currently support Intel compilers on any other machine for the software, compass that I'm trying to build Albany for.

[xylar]

@ikalash and @jewatkins, I could potentially try to add one of the machines you test with to mache and compass so you could test on your machine the same way we build on ours. That would get us a lot closer to testing the workflow we ultimately want to have work without interruptions. Is there a machine at Sandia that you test on that's also supported by E3SM? See the following file for all machines that E3SM runs on:
https://github.com/E3SM-Project/E3SM/blob/master/cime_config/machines/config_machines.xml

[bartgol]

snl-blake would be great, but I suspect that machine is not really maintained, since it was added by Micheal Deakin, who left SNL ~4yy ago, but is still listed as maintainer...

[jewatkins]

Okay, we can discuss more later if this isn't urgent. To me it looks like a compiler error, possibly due to the intel version which is not tested in Trilinos/Albany. I was thinking the easiest thing would be to try to update one of our internal builds to use the intel version that you're using and see if the compiler error comes up.

There's much we can do with a compiler error. We first have to identify the problem code. Then we'll either have tell the code owner to modify it (or we modify it and we have a special Trilinos version) or send a reproducer to the vendor and hope the issue is fixed in another release (and gets over to the target machines). But maybe it's as simple as turning something off like Luca suggests.

Skybridge is probably the closest thing to use for Sandia internal since we build e3sm and trilinos/albany there.

[jewatkins]

snl-blake would be great, but I suspect that machine is not really maintained, since it was added by Micheal Deakin, who left SNL ~4yy ago, but is still listed as maintainer...

Oh I did not see blake. It would be nice if we could revive that.

[xylar]

Well, keep me posted, I'm happy to help support whatever is practical.

[ikalash]

I suspect this is due to the compiler being so new. The newest intel compiler we are using in the Albany nightlies are 19.0.5. I could try building the code on one of our intel machines with the intel compiler that is there just as a sanity check and to test this theory. I'd try that on blake, as suggested above. Another thing to try would be to try building Trilinos the usual cmake way using the newer intel 20 compiler, but that's probably harder.

[ikalash]

Also, I have added this topic to the agenda for our 11/22 Albany meeting, just FYI.

[ikalash]

So I have an update on this - my apologies for the delay. I tried building albany using spack with intel on my workstation mockba using a sems module for the intel compiler, and the build completed. I used the following intel module there: module load sems-intel/2021.3. It therefore does not seem that intel is a fundamental problem for the spack build, newer versions of intel. I can try with some other versions of intel just as a sanity check. If that goes well, I would suggest someone try to build Trilinos from scratch using cmake on the problematic machines to see if the same issue is encountered.