Failing precompilation of HDF5_jll (with OpenMPI 4?)

[benegee]

CI currently fails when precompiling HDF5_jll

Failed to precompile HDF5_jll [0234f1f7-429e-5d53-9886-15a909be8d59] to "/home/runner/.julia/compiled/v1.10/HDF5_jll/jl_CgDWlD".
ERROR: LoadError: InitError: could not load library "/home/runner/.julia/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so"
/home/runner/.julia/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so: undefined symbol: ompi_instance_count

This was caused by the recent changes in OpenMPI_jll (JuliaPackaging/Yggdrasil@e449cb6), which declared the libmpi_mpifh library to be provided by OpenMPI_jll.

The actual cause however is precisely described here: JuliaPackaging/Yggdrasil#6893
We switch to system MPI, and also system HDF5, but HDF5_jll still gets pulled in and will instead load the MPI libraries provided by OpenMPI_jll. In particular it loads OpenMPI_jll's libmpi_mpifh.so, and cannot resolve the ompi_instance_count symbol (which comes with OpenMPI 5?).

Workarounds so far:

• Override the provided libmpi_mpifh.so library in LocalPreferences.toml

  [OpenMPI_jll]
  libmpi_mpifh_path = "/lib/x86_64-linux-gnu/libmpi_mpifh.so"
  

• Downgrade OpenMPI_jll to v4

@KerstinHartung

[sloede]

I don't understand: If HDF5 loads OpenMPI_jll, why would OpenMPI complain about a missing symbol in OpenMPI itself? Or is it that the system OpenMPI is loaded first, but then additional libraries are loaded within Julia and find the older system OpenMPI library (which does not contain the missing symbol)?

[benegee]

I have not checked, but that is what I assume. System's libmpi.so is loaded first, and OpenMPI_jll's libmpi.so therefore not loaded at all.

[sloede]

But why does it try to load libmpi_mpifh.so then - why isn't this already loaded from the system MPI?

[vchuravy]

@eschnett recently did JuliaPackaging/Yggdrasil#10884 maybe these things are connected?

[benegee]

MPI.jl, when set to system, loads system libmpi.so only.
And it does not load OpenMPI_jll (https://github.com/JuliaParallel/MPI.jl/blob/5fb0b9de5d270c203122b486736030ead31a93b0/src/api/api.jl#L11_L28)

System libhdf5.so also depends on libmpi.so only.

HDF5_jll however loads OpenMPI_jll, which loads all libraries that are declared as LibraryProduct (which includes libmpi_mpifh.so since the latest change).

At the moment I do not see why HDF_jll.jl gets loaded at all (https://github.com/JuliaIO/HDF5.jl/blob/b42c2a2b67e2686c5deab4dc19fbecf504e50654/src/api/api.jl#L10-L18).

[sloede]

At the moment I do not see why HDF_jll.jl gets loaded at all (https://github.com/JuliaIO/HDF5.jl/blob/b42c2a2b67e2686c5deab4dc19fbecf504e50654/src/api/api.jl#L10-L18).

Maybe create an MWE only with HDF5.jl (and possibly some explicit JLLs)? If the issue persists, I'd assume it's a bug that should be reported upstream.

[eschnett]

OpenMPI has one C and three Fortran libraries, and they do not all get loaded automatically when one of them is loaded. The recent fix I applied to OpenMPI_jll was to explicitly always load all 4 libraries when OpenMPI_jll is loaded.

The situation here might be similar. The system clearly knows which OpenMPI library to load, but maybe more than one need to be loaded and this isn't happening. Which OpenMPI library provides the missing symbol? Can you load it manually via LD_PRELOAD to see whether this solves the issue?

[sloede]

TL;DR

Downgrading OpenMPI_jll to v4 seems like the simplest fix for the time being

Longer version

@eschnett I think the problem is the API mismatch between MPI v4 and v5. MPI v5 introduced a number of new exported symbols to libmpi.so, which are used by the additional libmpixyz.so libraries. When we now load the system libmpi.so at MPI v4, OpenMPI_jll.jl will not load libmpi.so (since it's already there) but only the additional libmpixyz.so libraries from the JLL at MPI v5. Since the latter expect libmpi.so to have some v5-specific symbols, which it obviously does not have, we get the error above.

After talking to @giordano, we concluded that the easiest (downstream) solution for now likely is to restrict OpenMPI_jll to MPI v4 if these issues appear on a system. This also seems like the only solution if the additional libraries loaded by OpenMPI_jll.jl are not installed on a system.

Once most systems have updated their MPI installations to v5, we should actually have fewer problems due to the standardized ABI 🤞

ref: JuliaParallel/MPI.jl#835 (comment)

[eschnett]

When building against OpenMPI, the default is to build against OpenMPI 4.1.8, and to set compat bounds that allows running with OpenMPI 5 as well.

I don't understand your analysis: You assume that a library libmpi.so is already loaded – from where? From an MPI implementation different from OpenMPI_jll@5? That would be wildly inconsistent. The MUST be exactly one MPI implementation. If you want to load your own, different from OpenMPI_jll, then it's up to you to ensure that all necessary libraries are loaded. Apparently, OpenMPI ships many separate libraries, and there is no single library that would pull in all others, so you have to do that yourself (probably loading about 4 libraries manually). And no matter what you do, you CANNOT load any files from OpenMPI_jll if you loaded a system MPI library.

You also say that you want to use a system HDF5, and that for some reason HDF5_jll is still loading libraries. That seems to be the root cause of the problem – I don't think HDF5_jll should be loading any libraries in this case, and if it does, you need to meticulously tell it to load the right ones. Almost certainly these need to be other system libraries and not any libraries that are available as jll.

My recommendation is to use MPItrampoline if you want to use a system MPI library. Otherwise you need to be really, really careful to ensure that none of the OpenMPI_jll-provided libraries are loaded. This is difficult, tedious, and the best "error message" you can hope for is a segfault. If you want to use a system HDF5 library for some reason then let me know – maybe we could enable these features in HDF5_jll instead to save you the trouble?

The MPI ABI will specify certain things, but it will not specify library names, or how many libraries there are. It's unclear whether the ABI will even support old-style Fortran 77 MPI bindings that are still widely used (probably not). Unless MPI implementors work together there will be loopholes that can easily lead to problems such as the ones we see here.

[benegee]

Thanks @eschnett for the follow up! I really appreciate your help!

you CANNOT load any files from OpenMPI_jll if you loaded a system MPI library.

Exactly! This is what we would like to prevent. We would like to maintain a consistent environment based on the system MPI library.

Our setup looks like this (this is an ubuntu 24.04 workstation)

[HDF5]
libhdf5 = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so"
libhdf5_hl = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so"

[HDF5_jll]
libhdf5_path = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so"
libhdf5_hl_path = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so"

[MPIPreferences]
__clear__ = ["preloads_env_switch"]
_format = "1.0"
abi = "OpenMPI"
binary = "system"
cclibs = []
libmpi = "/lib/x86_64-linux-gnu/libmpi.so"
mpiexec = "mpiexec"
preloads = []

Starting from scratch with this LocalPreferences.toml

(juliatest) pkg> add MPI MPIPreferences
julia> using MPI
julia> MPI.versioninfo()
MPIPreferences:
  binary:  system
  abi:     OpenMPI
  libmpi:  /lib/x86_64-linux-gnu/libmpi.so
  mpiexec: mpiexec

Package versions
  MPI.jl:             0.20.22
  MPIPreferences.jl:  0.1.11

Library information:
  libmpi:  /lib/x86_64-linux-gnu/libmpi.so
  libmpi dlpath:  /lib/x86_64-linux-gnu/libmpi.so
  MPI version:  3.1.0
  Library version:
    Open MPI v4.1.6, package: Debian OpenMPI, ident: 4.1.6, repo rev: v4.1.6, Sep 30, 2023

Adding HDF5.jl now results in the reported error ( undefined symbol: ompi_instance_count) when HDF5_jll.jl is being precompiled.

To reproduce the clash manually:

• Loading the OpenMPI_jll@5 libraries works

  julia> Libdl.dlopen("./julia-depot/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi.so")
  Ptr{Nothing} @0x000000003e008d60
  julia> Libdl.dlopen("./julia-depot/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so")
  Ptr{Nothing} @0x000000003dbd73c0
  

• Loading the system MPI library and then the OpenMPI_jll@5 libraries gives the error

  julia> Libdl.dlopen("/lib/x86_64-linux-gnu/libmpi.so")
  Ptr{Nothing} @0x0000000026622270
  julia> Libdl.dlopen("./julia-depot/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so")
  ERROR: could not load library "./julia-depot/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so"
  ./julia-depot/artifacts/689e8d7a066e74f34108119415bd9e46a2f8b168/lib/libmpi_mpifh.so: undefined symbol: ompi_instance_count
  

• Downgrading to OpenMPI_jll@4, loading the system MPI library and then OpenMPI_jll@4 libraries works again

  julia> Libdl.dlopen("/lib/x86_64-linux-gnu/libmpi.so")
  Ptr{Nothing} @0x0000000027346880
  julia> Libdl.dlopen("./julia-depot/artifacts/d2e95c5d3bdb3489102b88bca6b38de65f82db3e/lib/libmpi_mpifh.so")
  Ptr{Nothing} @0x00000000275c6cd0
  

Coming back to our setup:

1. Adding MPI triggers precompilation of OpenMPI_jll among others, which works
2. Adding HDF5 triggers precompilation of HDF5_jll, which fails
3. Using HDF5 afterwards works

[vchuravy]

Adding HDF5.jl now results in the reported error ( undefined symbol: ompi_instance_count) when HDF5_jll.jl is being precompiled.

I think part of the problem here is that we are trying to support more than we actually can...

When I wrote MPIPreferences.jl, I was worried about a situation where, the user would use the system libraries, but wouldn't be diligently setting library overrides for all possible MPI dependencies (I felt that would be untracktable).

So back then I decided to only split on the ABI, not on the fact if we are using a system library or not.

So HDF5_jll actually performs a using OpenMPI_jll iff the ABI is set to OpenMPI

julia> using HDF5_jll

julia> HDF5_jll.host_platform
Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.11.4, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30, mpi=openmpi}

(HDF5_loading) pkg> why OpenMPI_jll
  HDF5_jll → OpenMPI_jll

(The logic is in https://github.com/JuliaPackaging/Yggdrasil/blob/master/platforms/mpi.jl, where we push a Dependency on the OpenMPI_jll)

Essentially, I forgot the step where using the system MPI should block OpenMPI_jll from loading its libraries.
I am not a 100% how this worked previously, but it seems dependent on the order of operations.
Likely this worked due to MPI.jl being loaded first, which will dlopen the system libmpi, therefore blocking OpenMPI_jll from loading its libmpi.

After JuliaPackaging/Yggdrasil@e449cb6 we had more than just libmpi being loaded when OpenMPI_jll is implicitly loaded through HDF5_jll.

TL;DR: My assumption was that it would be okay to load libraries provided from Yggdrasil that match the ABI of the system.

So how can we fix this?

1. Add a "system" ABI that just doesn't try to use any provided libraries

• UX issues for users since they will need to provide explicit preference files or similar for all JLLs depending on MPI (not just the ones they care about)
• Implementation issue: JLLWrappers currently doesn't generate platform target files to load, so actually there are no preferences that we could set. This is similar to trying to use a JLL on a platform it wasn't build for, one can't provide preferences to custom builds.

2. Somehow add special handling for OpenMPI_jll & co to early exit their init functions when system ABI is used.
   • Maybe we can cheat this by providing a platform tag just for the OpenMPI_jll & co libraries?
   • This will potentially break other JLLs, but loudly

[vchuravy]

Turns out 2. is relatively straigforward to implement JuliaPackaging/Yggdrasil#10977