ifort builds fail on gaea c5

[AlexanderRichert-NOAA]

Describe the bug
On the head of develop, the error loading plugin: libimf.so: cannot open shared object file: No such file or directory ifort error is occurring.

To Reproduce
Clone head of develop, create an empty environment, and install, e.g., bufr or openblas.

Expected behavior

System:
gaea c5 (gaea58)

[climbfuji]

isn't that a case for the usual compiler - env - prepend-path workaround we've been using for years on several machines?

[AlexanderRichert-NOAA]

Yep

[RatkoVasic-NOAA]

@AlexanderRichert-NOAA, what change is needed, how it was done before?

[AlexanderRichert-NOAA]

See for example discover and derecho's compilers.yaml, where for Intel, there's an LD_LIBRARY_PATH prepend_path setting, which uses a path ending with 'intel64_lin' (which contains libimf.so). @climbfuji do you remember if there's a reason we didn't also add it to extra_rpaths?

[climbfuji]

I don't remember. Presumably because it works ;-)

[AlexanderRichert-NOAA]

I also ran into an issue on gaea just now where it wants libfabric, even though it's a serial code. Seemingly relatedly, I can see -lmpifort_intel in the icc arguments, I'm not sure where it's coming from.

[climbfuji]

cray compiler wrappers maybe?

[AlexanderRichert-NOAA]

@RatkoVasic-NOAA ?

[RatkoVasic-NOAA]

This is the message I got:

==> Error: CompilerAccessError: Compiler 'intel@=2023.1.0' has executables that are missing or are not executable: ['CC', 'ftn', 'ftn']
==> Error: bufr-12.1.0-pj64ai6phu2k676naq4efbb2xvbntfnc: Package was not installed
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

I checked out current develop on gaea-c5, created empty environment and:
spack add bufr@12.1.0
spack install bufr@12.1.0

[RatkoVasic-NOAA]

When I changed:

      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn

to:

      cc: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/icc
      cxx: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/icpc
      f77: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort
      fc: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort

in configs/sites/tier1/gaea-c5/compilers.yaml
And 2023.1.0 to 2023.2.0
in configs/sites/tier1/gaea-c5/compilers.yaml and configs/sites/tier1/gaea-c5/packages.yaml
It started compiling... it is very slow, I'll check in the morning how it went.

[climbfuji]

If you replace the Cray compiler wrappers with the direct compiler binaries, then you

• need to also toggle the wrappers variant for cray-mpich
• step into uncharted terrain, see Investigate removal of Cray compiler wrappers for Narwhal, Gaea C5 #771; this may or may not work, depending on the version of spack, craype, ...

That being said, in the long term this is what we (and also the spack developers and HPE want), I am just not sure if Gaea C5 is new enough for it.

[RatkoVasic-NOAA]

It did work this way (/gpfs/f5/epic/scratch/Ratko.Vasic/SS-1362/spack-stack/envs/develop/install/intel/2023.2.0/).
But gaea-c5 was acting weird, even though I installed in background, I was loosing connection (I guess login nodes were going up and down) and had to restart installation several times.

[climbfuji]

I wouldn't be surprised if it doesn't run correctly, though. Let's see!