SCALAPACK32: New version 2.2.2

[eschnett]

No description provided.

[amontoison]

@eschnett What do you think of using LBT_jll.jl instead of OpenBLAS32_jll.jl?
I wanted to do that since a long time.

We need to update MUMPS.jl and PETsc.jl to load a BLAS / LAPACK but that almost the only dependent packages.

[eschnett]

Sounds good (I don't know what that entails). I mostly wanted to clean up how MPI is handled. Do you want to make the respective changes?

[eschnett]

@amontoison Are you still planning to introduce libblastrampoline?

[amontoison]

Yes but we need to load a LP64 BLAS / LAPACK for high-level packages like MUMPS.jl / PETSc.jl in this case and I would like to do that we merge the PR such that we don't break them.

[eschnett]

Since this is an incompatible change I recommend changing the major version number of this package (SCALAPACK32) to prevent accidental breakage. That is, I recommend the following:

• merge this PR as-is
• create a release of SCALAPACK32 with a different major version number
• work on the dependencies which can choose which major version of SCALAPACK32 to choose
• in the end, once all dependencies can use the new version of SCALAPACK32, things will migrate automatically

This uses major version numbers in the way they are intended by semver. There is no juggling to make sure everything happens simultaneously enough, and there is no chance of accidentally breaking anything.

[amontoison]

@eschnett If we change the major version number of SCALAPACK32_jll, it will be the same issue because the other JLL compiled with it like MUMPS_jll or PETSc_jll don't have a compat entry on it.

[amontoison]

@eschnett I opened a PR in the general registry to update the needed compat entry on SCALAPACK32_jll and ensure that we don't break anybody with this new release: JuliaRegistries/General#140191

As explained in CONTRIBUTING.md, the next step is to update the compat entry on SCALAPACK32_jll.jl here for the three concerned packages (MUMPS_jll, QuantumEspresso_jll and PETSc_jll (files build_tarballs.jl).

Then it will be safe to merge this PR.

[eschnett]

@amontoison I see that you do not want to introduce a new major version for this incompatible change. I have been there in the past, with e.g. HDF5 or other packages making incompatible changes without increasing the major version number. This leads to a complicated mess, and any mistake will lead to segfaults at end users and will be difficult to track down. There is a long list of problematic packages here, and if we were to make an incompatible change here, then we would need to add SCALAPACK32 to this list.

The upper compat bound for all packages using SCALAPACK or SCALAPACK32 should be -2, allowing future releases which increase the minor version number.

We should also update the source of these packages (PETSc etc.) to introduce a respective compat bound in their build script at the same time, so that the package source and the registry match.

[amontoison]

@amontoison I see that you do not want to introduce a new major version for this incompatible change. I have been there in the past, with e.g. HDF5 or other packages making incompatible changes without increasing the major version number. This leads to a complicated mess, and any mistake will lead to segfaults at end users and will be difficult to track down. There is a long list of problematic packages here, and if we were to make an incompatible change here, then we would need to add SCALAPACK32 to this list.

The issue with a bump of the major version of the JLL is that it doesn't fix any of the compatibility issue because we never set any compat entry on SCALAPACK32_jll.jl in the other build_tarballs.jl.
So I prefer to keep sync with the official version number of SCALAPACK.
But I totally agree to add a note in the list of problematic packages!

The upper compat bound for all packages using SCALAPACK or SCALAPACK32 should be -2, allowing future releases which increase the minor version number.

I agree with that for SCALAPACK because LBT automatically forward the symbols to OpenBLAS_jll.jl, which is shipped with Julia. So it is totally non-breaking.

The issue is with SCALAPACK32 because we need to provide a LP64 BLAS / LAPACK with something like this:
https://github.com/jump-dev/Ipopt.jl/blob/master/src/Ipopt.jl#L15-L18
So we need first to add a compat entry 2.1.0-2.2.1 such we don't break the current users (it is what I did in the general registry) and the next time that we compile a new release of PETSc or MUMPS, we can change it for 2.2.2-2 if the high-level interface load the LP64 backends.
I maintain MUMPS.jl so it is easy for me. We really need to be careful for PETSc.jl.

We should also update the source of these packages (PETSc etc.) to introduce a respective compat bound in their build script at the same time, so that the package source and the registry match.

It is what I explained in my previous message but we need a merge in the general registry first.

[imciner2]

In general, we haven't made JLLs that need LP32 BLAS link against LBT because there is no guarantee there is a backing library loaded, and so the JLL on its own would essentially be broken when it is installed. While users could manually add OpenBLAS32 (or their favorite backend) to LBT, they might not realize they need to do that.

IMO, before we link LP64 consumer libraries against LBT, we should build a way for the JLLs to selectively load OpenBLAS32 at runtime. We could make it more generic actually and have a flag for build_tarballs that says if it wants a BLAS backend - and which one. Then have BinaryBuilder automatically add the dependency to LBT and the appropriate version of OpenBLAS, then have the JLLWrappers package do the forwarding if there are no forwards already.

[giordano]

In general, we haven't made JLLs that need LP32 BLAS link against LBT

There's actually some, like ipopt, but consumer packages need to do something like https://github.com/jump-dev/Ipopt.jl/blob/480c9ecb0755b435c60aa5a9381a9e8e0b219c70/src/Ipopt.jl#L14-L20, otherwise everything goes boom.

[imciner2]

In general, we haven't made JLLs that need LP32 BLAS link against LBT

There's actually some, like ipopt, but consumer packages need to do something like jump-dev/Ipopt.jl@480c9ec/src/Ipopt.jl#L14-L20, otherwise everything goes boom.

Yes, although I think the "boom" factor with ipopt is lower than with something like SCALAPACK32, because ipopt generally has fewer consumers but these base libraries could be used by many (already there appear to be 3 JLLs that reference this), and so then any consumer of those JLLs would need to have the loading logic in it - which just becomes a very large "boom" surface.

[ViralBShah]

It would be easiest if openblas could compile LP64 shims that just forward to ILP64 so that we can have both by default in Julia. Then LBT can have forward for both as well and a lot of stuff would simplify.

[giordano]

That doesn't address the problem that we still need to have an LP64 openblas around.

[ViralBShah]

My mental model was if openblas has both lp64 and ilp64, then we are effectively shipping both BLAS.

[amontoison]

In general, we haven't made JLLs that need LP32 BLAS link against LBT because there is no guarantee there is a backing library loaded, and so the JLL on its own would essentially be broken when it is installed. While users could manually add OpenBLAS32 (or their favorite backend) to LBT, they might not realize they need to do that.

IMO, before we link LP64 consumer libraries against LBT, we should build a way for the JLLs to selectively load OpenBLAS32 at runtime. We could make it more generic actually and have a flag for build_tarballs that says if it wants a BLAS backend - and which one. Then have BinaryBuilder automatically add the dependency to LBT and the appropriate version of OpenBLAS, then have the JLLWrappers package do the forwarding if there are no forwards already.

@imciner2 I disagree that we generally don't compile LP64 consumer libraries against LBT.
Since LBT 5.4.0 / Julia 1.9, we try to use LBT is a few LP64 consumer libraries:

• Ipopt
• Uno
• MUMPS_seq
• HSL
• SPRAL
• qr_mumps
• PROPACK
• COIN-OR packages

[ViralBShah]

I believe the main issue is that LBT forwards have to be set up, and there isn't a convenient way to do that right now. For example, in Sundials.jl I just linked against OpenBLAS32_jll. If I link against LBT, I still need to make sure that an LP64 BLAS is available, then do the lbt forwards - might as well link to openblas32 directly.

My latest thought thus was to include LP64 APIs in openblas that forward to the ILP64 ones - there by allowing us to ship a fully configured LBT+openblas with Julia.

[amontoison]

In general, we haven't made JLLs that need LP32 BLAS link against LBT

There's actually some, like ipopt, but consumer packages need to do something like jump-dev/Ipopt.jl@480c9ec/src/Ipopt.jl#L14-L20, otherwise everything goes boom.

Yes, although I think the "boom" factor with ipopt is lower than with something like SCALAPACK32, because ipopt generally has fewer consumers but these base libraries could be used by many (already there appear to be 3 JLLs that reference this), and so then any consumer of those JLLs would need to have the loading logic in it - which just becomes a very large "boom" surface.

What is your source?? Ipopt_jll is used by WAY more people and packages than SCALAPACK32_jll.
Just look at this:
(https://juliahub.com/ui/Packages/General/Ipopt_jll/300.1400.1700+0#dependents).
Everybody relies on Ipopt compiled with LBT.
And many packages /JLL need Ipopt_jll directly.
While here, 99.9% of the people will use the high-level interfaces MUMPS.jl and PETSc.jl.

SCALAPACK is quite specific and not needed very often. (Only three JLL need it). To the best of my knowledge, I only know MUMPS and PETSc. I discovered the third one here.
I don't see the "boom" factor for this package.

Prove me that I am wrong @imciner2 but I strongly disagree with your last two points:

• LBT is not used often for LP64 libraries
• boom factor for SCALAPACK32 is bigger then Ipopt

[ViralBShah]

@amontoison Would you recommend Sundials follow the same route? Link to LBT for LP64, depend on openblas32_jll and then set up the lbt_forward in the init?

Using LBT while simultaneously depending on OpenBLAS32 and for every package then to set up the forward feels a little awkward (but naturally necessary currently).

[ViralBShah]

Good to merge?

[amontoison]

Good to merge?

Not yet, I need to update the compat entries on JLL of SCALAPACK32_jll in the build_tarballs.jl of PETSC_jll, MUMPS_jll, etc.. I will do that now.

[amontoison]

@amontoison Would you recommend Sundials follow the same route? Link to LBT for LP64, depend on openblas32_jll and then set up the lbt_forward in the init?

Using LBT while simultaneously depending on OpenBLAS32 and for every package then to set up the forward feels a little awkward (but naturally necessary currently).

@ViralBShah Yes, I think it is a good idea!
It could lead to a very nice speed-up: jump-dev/Ipopt.jl#360 (comment)

[ViralBShah]

@amontoison Would you recommend Sundials follow the same route? Link to LBT for LP64, depend on openblas32_jll and then set up the lbt_forward in the init?
Using LBT while simultaneously depending on OpenBLAS32 and for every package then to set up the forward feels a little awkward (but naturally necessary currently).

@ViralBShah Yes, I think it is a good idea! It could lead to a very nice speed-up: jump-dev/Ipopt.jl#360 (comment)

Ok thanks. I will do that when the next sundials release is out.