You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HiPO build ignores user choice of BLAS on Apple #3160
We can't upgrade to HiGHS v1.15.1 in Uno because the static compilation of HiGHS is now always compiled by expecting an OpenBLAS backend.
We don't see any error during the compilation of HiGHS because we never call the linker but when we link it in Uno, it fails because it requires some symbols of OpenBLAS.
I see two issues here:
1- The BLAS backend is ignored.
Even if we don't link to the BLAS library in a static compilation, it gives a hint of what additional symbols can be expected (or not) after by the user to provide.
If we provide a LBT library at the compilation of HiGHS, we don't want to require AppleAccelerate, intel MKL or OpenBLAS threading symbols at runtime.
2- If the user doesn't provide a valid BLAS library or no BLAS library is found, but we do a static compilation, I suggest to not require any threading symbols such that any BLAS backend can be still linked (reference BLAS, MKL, LBT, AOCL, ...). It will be less efficient but it is a very strong assumption to expect OpenBLAS symbols at runtime if not provided at compilation time.
I suggest to add a CI job where you just use the reference BLAS or another BLAS backend than OpenBLAS.
mkdir highs_reproducer
cd highs_reproducer
mkdir installation
git clone https://github.com/Reference-LAPACK/lapack.git
cd lapack
git checkout v3.12.1
mkdir build
cd build
cmake .. \
-DCBLAS=ON \
-DLAPACKE=OFF \
-DCMAKE_INSTALL_PREFIX=../../installation \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DBUILD_INDEX64_EXT_API=OFF \
-DTEST_FORTRAN_COMPILER=OFF
make -j32
make install
cd ../..
git clone https://github.com/ERGO-Code/HiGHS.git
cd HiGHS
git checkout v1.15.1
# git checkout v1.15.0
mkdir build
cd build
cmake .. \
-DCMAKE_INSTALL_PREFIX=../../installation \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DZLIB=OFF \
-DHIPO=ON \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTING=OFF \
-DBUILD_CXX_EXE=OFF \
-DBLAS_LIBRARIES=../../installation/lib/libblas.a \
-DBUILD_SHARED_EXTRAS_LIB=OFF \
-DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j32
make install
cd ../../installation/lib
g++ -shared -Wl,--whole-archive libhighs.a libhighs_extras.a -Wl,--no-whole-archive -Wl,--no-undefined -lblas -o libhighs.so
We have the following error only on v1.15.1:
/usr/bin/ld: libhighs_extras.a(myblas.cpp.o): infunction`highs_openblas_set_num_threads':
myblas.cpp:(.text+0x1): undefined reference to `openblas_set_num_threads'collect2: error: ld returned 1 exit status
Update: For the comment 2- in my previous message, it seems that HiGHS is already doing the right thing:
mkdir highs_reproducer2
cd highs_reproducer2
mkdir installation
git clone https://github.com/Reference-LAPACK/lapack.git
cd lapack
git checkout v3.12.1
mkdir build
cd build
cmake .. \
-DCBLAS=ON \
-DLAPACKE=OFF \
-DCMAKE_INSTALL_PREFIX=../../installation \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DBUILD_INDEX64_EXT_API=OFF \
-DTEST_FORTRAN_COMPILER=OFF
make -j32
make install
cd ../..
git clone https://github.com/ERGO-Code/HiGHS.git
cd HiGHS
git checkout v1.15.1
mkdir build
cd build
cmake .. \
-DCMAKE_INSTALL_PREFIX=../../installation \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DZLIB=OFF \
-DHIPO=ON \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTING=OFF \
-DBUILD_CXX_EXE=OFF \
-DBLAS_LIBRARIES= \
-DBUILD_SHARED_EXTRAS_LIB=OFF \
-DCMAKE_POSITION_INDEPENDENT_CODE=ON
make -j32
make install
cd ../../installation/lib
g++ -shared -Wl,--whole-archive libhighs.a libhighs_extras.a -Wl,--no-whole-archive -Wl,--no-undefined -lblas -o libhighs.so
I don't have any linker error.
So the issue in HiGHS is when we explicitly provide the path of the BLAS library.
@cvanaret You can use -DBLAS_LIBRARIES= \ (or just not pass at all this option) as a workaround for UnoUtils_jll.jl, it seems to not impose the OpenBLAS backend.
It is just a hacky solution...
Yes, at the moment for HiPO only OpenBLAS is supported on Windows and linux. So it is expected that you will get a failure with reference blas, and we want to discourage users from using it at all because of its inferior performance.
We do intend to support MKL in the future but have more work to do there.
So, as far as LBT is concerned, we only really want to use Accelerate on apple and openblas on win/linux
Yes, at the moment for HiPO only OpenBLAS is supported on Windows and linux. So it is expected that you will get a failure with reference blas, and we want to discourage users from using it at all because of its inferior performance.
Sorry @galabovaa but it don't unserstand your comment.
It was working with HiGHS v1.15.0 and you remove the support in a patch release?
Did you documented somewhere that only this BLAS backend is supported?
If we rely on the CMake detection, HiPO is compiled with reference BLAS and it is working with v1.15.1.
Like it was working with v1.15.0.
The issue is when we provide the path explicitly.
It seems completly orthogonal to the support or not of the BLAS backend.
If it was not supported, then why do you finally support it by compiling it without the OpenBLAS symbols in the library?
I am lost here.
I see how it is slightly unrelated to the issue here, apologies for my unclear comment.
The drop was in the release notes of 1.15.0 and I forgot to update the documentation website. Whilst reference blas may still work under some configurations, it will not always with dynamic loading of the shared libraries now that we have highs_extras. I had a CI linux test that was failing with reference blas at 1.15.0 or 1.15.1.
It is still better to use openblas for Uno, and I suspected that was a possible reason Uno was failing to build with the path to the library. That may not be the case. The issue here no. 3160 occurs on macOS only, and you are posting Uno build logs and a reproducer on Linux. That should point to a related but different issue, and I wanted to note it is really better to use OpenBLAS.
@odow The core issue is that HiGHS links against the old Apple Accelerate interface.
The ~15-year-old one, frozen at LAPACK 3.2.1 and not aligned with Netlib, with long-standing known correctness quirks in its BLAS.
That's completely different from the modern interface Apple shipped in macOS >= 13.3 (LAPACK 3.9.1, Netlib-aligned, actually maintained and tested, it's the one we forward to through LBT).
Concretely, HiGHS declares its own un-suffixed cblas_* prototypes and links -framework Accelerate without ACCELERATE_NEW_LAPACK, so it resolves to the legacy symbols.
So the "use the vendor BLAS" argument is a bit shaky here: you're getting the worst Accelerate, not a robust one.
Switching is straightforward, the load-bearing part is including <Accelerate/Accelerate.h> with ACCELERATE_NEW_LAPACK defined on Apple (the macro alone does nothing without Apple's header,
since it works via $NEWLAPACK asm labels; it means bypassing the hand-rolled mycblas.h on macOS).
Sticking with the old Accelerate is a real regression vs. the new one (in Julia), which fixes bugs that have been in the legacy BLAS for years.
LBT is forwarding to the new version: #3160
Thanks @galabovaa, and to be clear, I really appreciate you and the whole HiGHS team,
so I will do my best to be extra diplomatic here.
I'm just struggling to follow some of the arguments.
To me this reads as a build-system regression, not a BLAS-backend question.
The BLAS/CBLAS API has been a stable standard for decades, and for what HiGHS actually calls
(double-precision CBLAS, LP64) reference BLAS, OpenBLAS, BLIS and AOCL are interchangeable down to the integer-width ABI.
The only OpenBLAS-specific touch is openblas_set_num_threads, which HiGHS already wraps as a no-op when OpenBLAS isn't used.
It can't be what gates backend support.
And thread control is orthogonal: every backend drives it through its own env vars (OPENBLAS_NUM_THREADS, OMP_NUM_THREADS, BLIS_NUM_THREADS, ...) or dedicated symbols, independent of which library you link.
So "reference BLAS isn't supported" doesn't really hold: the failure isn't in the numerics, it's that FindHipoDeps.cmake (since #3031) overrides the user's BLAS choice, hardcoded BLA_VENDOR Apple on macOS (that's #3160), and an OpenBLAS-assuming symbol on Linux (my reproducer).
Same root cause, the user's selection is bypassed.
It is just a different symptom per platform.
The part that worries me most: this shipped in a patch release (1.15.0 → 1.15.1) and broke downstream packaging (4th time Uno breaks because of a HiGHS build change).
Dropping a previously-working configuration is a backwards-incompatible change, per SemVer that not belongs to patch bump.
Happy to keep the macOS and Linux tickets separate for tracking, but I'd push back on treating them as unrelated: fix
the CMake layer so it respects the user-provided BLAS and both go away.
Conclusion, we need to compile HiGHS without HiPO now in Uno. :(
Thank you for your comments, @amontoison and @odow. I will fix the CMake wrapper issues and old Apple Accelerate issue. I will also add error messages if reference blas is used, to avoid any further confusion. There really is no point in including HiPO if its performance will be hindered by libblas.
Please correct me if I am wrong, but I thought we added https://github.com/ERGO-Code/HiGHS/actions/workflows/uno-reverse-ci.yml to avoid breaking the Uno build again in the future. Is the current break missing from the workflow tests? Should we consider adding, to test the upcoming fix, and so you could more reliably use HiPO in Uno?
Thanks for looking into this @galabovaa.
It is natural that we expect different things from HiPO at the moment: you're after maximum performance on large instances, and we care about flexibility and reproducibility.
I agree with Oscar and Alexis that ideally, the user should pick their BLAS. Since we plan to add HiPO to our test infrastructure and ship it in our wheels and JLLs, we need a build that behaves predictably across platforms: we need to know exactly which BLAS is linked, that it's the same on every machine, and that we can swap it if necessary.
I also understand that you're running HiGHS and HiPO on massive instances and that you want to position them well in benchmarks, and at that scale the choice of BLAS is not a minor detail.
Perhaps this would work for both sides: keep the fast BLAS as the default, and give us a documented flag to override it.
It's not just about getting maximum performance. We fixed the BLAS because some users - who just want local results - and are less enlightened than yous were using inappropriate BLAS and then complained that they were getting poor parallel performance.
We're happy to give folks like you the freedom to choose BLAS for your own reasons, although it's unfortunate that this means inferior performance from HIPO. Please make this clear in your documentation
@amontoison and @filikat I also updated mycblas to use #define ACCELERATE_NEW_LAPACK if Apple Accelerate is used at compile time. I need to still think about how to do that at runtime. I also added the extras dll to the cmake install, it was missing.
@amontoison and @cvanaret Could you possibly double check if this resolves the Uno build issues?
Activity
amontoison commentedon Aug 23, 2026
I think that it is ignored on all platforms: cvanaret/Uno#869
We can't upgrade to HiGHS v1.15.1 in Uno because the static compilation of HiGHS is now always compiled by expecting an OpenBLAS backend.
We don't see any error during the compilation of HiGHS because we never call the linker but when we link it in Uno, it fails because it requires some symbols of OpenBLAS.
I see two issues here:
1- The BLAS backend is ignored.
Even if we don't link to the BLAS library in a static compilation, it gives a hint of what additional symbols can be expected (or not) after by the user to provide.
If we provide a LBT library at the compilation of HiGHS, we don't want to require AppleAccelerate, intel MKL or OpenBLAS threading symbols at runtime.
2- If the user doesn't provide a valid BLAS library or no BLAS library is found, but we do a static compilation, I suggest to not require any threading symbols such that any BLAS backend can be still linked (reference BLAS, MKL, LBT, AOCL, ...). It will be less efficient but it is a very strong assumption to expect OpenBLAS symbols at runtime if not provided at compilation time.
I suggest to add a CI job where you just use the reference BLAS or another BLAS backend than OpenBLAS.
cc @cvanaret
amontoison commentedon Aug 23, 2026
I prepared a reproducer on linux:
We have the following error only on
v1.15.1:amontoison commentedon Aug 23, 2026
Update: For the comment
2-in my previous message, it seems that HiGHS is already doing the right thing:I don't have any linker error.
So the issue in HiGHS is when we explicitly provide the path of the BLAS library.
@cvanaret You can use
-DBLAS_LIBRARIES= \(or just not pass at all this option) as a workaround forUnoUtils_jll.jl, it seems to not impose the OpenBLAS backend.It is just a hacky solution...
cvanaret commentedon Aug 24, 2026
I gave
-DBLAS_LIBRARIES= \a shot, but got:Same thing when I don't pass the option.
galabovaa commentedon Aug 24, 2026
Yes, at the moment for HiPO only OpenBLAS is supported on Windows and linux. So it is expected that you will get a failure with reference blas, and we want to discourage users from using it at all because of its inferior performance.
We do intend to support MKL in the future but have more work to do there.
So, as far as LBT is concerned, we only really want to use Accelerate on apple and openblas on win/linux
cc @mathgeekcoder
galabovaa commentedon Aug 24, 2026
@amontoison and @cvanaret you should also try using -DBLA_VENDOR=OpenBLAS for win and linux, as we have done in vcpkg
amontoison commentedon Aug 24, 2026
Sorry @galabovaa but it don't unserstand your comment.
It was working with HiGHS v1.15.0 and you remove the support in a patch release?
Did you documented somewhere that only this BLAS backend is supported?
If we rely on the CMake detection, HiPO is compiled with reference BLAS and it is working with v1.15.1.
Like it was working with v1.15.0.
The issue is when we provide the path explicitly.
It seems completly orthogonal to the support or not of the BLAS backend.
If it was not supported, then why do you finally support it by compiling it without the OpenBLAS symbols in the library?
I am lost here.
galabovaa commentedon Aug 24, 2026
I see how it is slightly unrelated to the issue here, apologies for my unclear comment.
The drop was in the release notes of 1.15.0 and I forgot to update the documentation website. Whilst reference blas may still work under some configurations, it will not always with dynamic loading of the shared libraries now that we have highs_extras. I had a CI linux test that was failing with reference blas at 1.15.0 or 1.15.1.
It is still better to use openblas for Uno, and I suspected that was a possible reason Uno was failing to build with the path to the library. That may not be the case. The issue here no. 3160 occurs on macOS only, and you are posting Uno build logs and a reproducer on Linux. That should point to a related but different issue, and I wanted to note it is really better to use OpenBLAS.
odow commentedon Aug 24, 2026
I'm going to push back a bit here. BLAS is a standard component. You should let the user choose which BLAS to use.
If
BLA_VENDORorBLAS_LIBRARIESis set, we should respect them.On the Julia side, we don't need any special support for MKL because libblastrampoline already supports that.
If we want special support for accelerate and openblas that's fine. But we should also respect someone opting out of that.
I thought I had change the Julia build job to use libblastrampoline, but I guess not:
HiGHS/.github/julia/build_tarballs.jl
Line 38 in 73cac48
odow commentedon Aug 24, 2026
Ah, I did on
latest:HiGHS/.github/julia/build_tarballs.jl
Line 36 in f614a82
but
masteris different?odow commentedon Aug 25, 2026
@amontoison here's a test: jump-dev/HiGHS.jl#350
We use blastrampoline on Windows and Linux, but not on macOS. It should be sufficient to just set
BLA_VENDOR = libblastrampoline.You can see it in the latest CI job on
latest:The issue is this line
HiGHS/cmake/FindHipoDeps.cmake
Lines 432 to 433 in f614a82
which links accelerate irrespective of the user's options.
amontoison commentedon Aug 25, 2026
@odow The core issue is that HiGHS links against the old Apple Accelerate interface.
The ~15-year-old one, frozen at LAPACK 3.2.1 and not aligned with Netlib, with long-standing known correctness quirks in its BLAS.
That's completely different from the modern interface Apple shipped in macOS >= 13.3 (LAPACK 3.9.1, Netlib-aligned, actually maintained and tested, it's the one we forward to through LBT).
Concretely, HiGHS declares its own un-suffixed
cblas_*prototypes and links-framework AcceleratewithoutACCELERATE_NEW_LAPACK, so it resolves to the legacy symbols.So the "use the vendor BLAS" argument is a bit shaky here: you're getting the worst Accelerate, not a robust one.
Switching is straightforward, the load-bearing part is
including <Accelerate/Accelerate.h>withACCELERATE_NEW_LAPACKdefined on Apple (the macro alone does nothing without Apple's header,since it works via
$NEWLAPACKasm labels; it means bypassing the hand-rolledmycblas.hon macOS).Sticking with the old Accelerate is a real regression vs. the new one (in Julia), which fixes bugs that have been in the legacy BLAS for years.
LBT is forwarding to the new version: #3160
amontoison commentedon Aug 25, 2026
Thanks @galabovaa, and to be clear, I really appreciate you and the whole HiGHS team,
so I will do my best to be extra diplomatic here.
I'm just struggling to follow some of the arguments.
To me this reads as a build-system regression, not a BLAS-backend question.
The BLAS/CBLAS API has been a stable standard for decades, and for what HiGHS actually calls
(double-precision CBLAS, LP64) reference BLAS, OpenBLAS, BLIS and AOCL are interchangeable down to the integer-width ABI.
The only OpenBLAS-specific touch is
openblas_set_num_threads, which HiGHS already wraps as a no-op when OpenBLAS isn't used.It can't be what gates backend support.
And thread control is orthogonal: every backend drives it through its own env vars (
OPENBLAS_NUM_THREADS,OMP_NUM_THREADS,BLIS_NUM_THREADS, ...) or dedicated symbols, independent of which library you link.So "reference BLAS isn't supported" doesn't really hold: the failure isn't in the numerics, it's that
FindHipoDeps.cmake(since #3031) overrides the user's BLAS choice, hardcodedBLA_VENDORApple on macOS (that's #3160), and an OpenBLAS-assuming symbol on Linux (my reproducer).Same root cause, the user's selection is bypassed.
It is just a different symptom per platform.
The part that worries me most: this shipped in a patch release (1.15.0 → 1.15.1) and broke downstream packaging (4th time Uno breaks because of a HiGHS build change).
Dropping a previously-working configuration is a backwards-incompatible change, per SemVer that not belongs to patch bump.
Happy to keep the macOS and Linux tickets separate for tracking, but I'd push back on treating them as unrelated: fix
the CMake layer so it respects the user-provided BLAS and both go away.
Conclusion, we need to compile HiGHS without HiPO now in Uno. :(
galabovaa commentedon Aug 26, 2026
Thank you for your comments, @amontoison and @odow. I will fix the CMake wrapper issues and old Apple Accelerate issue. I will also add error messages if reference blas is used, to avoid any further confusion. There really is no point in including HiPO if its performance will be hindered by libblas.
Please correct me if I am wrong, but I thought we added https://github.com/ERGO-Code/HiGHS/actions/workflows/uno-reverse-ci.yml to avoid breaking the Uno build again in the future. Is the current break missing from the workflow tests? Should we consider adding, to test the upcoming fix, and so you could more reliably use HiPO in Uno?
cvanaret commentedon Aug 27, 2026
Thanks for looking into this @galabovaa.
It is natural that we expect different things from HiPO at the moment: you're after maximum performance on large instances, and we care about flexibility and reproducibility.
I agree with Oscar and Alexis that ideally, the user should pick their BLAS. Since we plan to add HiPO to our test infrastructure and ship it in our wheels and JLLs, we need a build that behaves predictably across platforms: we need to know exactly which BLAS is linked, that it's the same on every machine, and that we can swap it if necessary.
I also understand that you're running HiGHS and HiPO on massive instances and that you want to position them well in benchmarks, and at that scale the choice of BLAS is not a minor detail.
Perhaps this would work for both sides: keep the fast BLAS as the default, and give us a documented flag to override it.
jajhall commentedon Aug 27, 2026
It's not just about getting maximum performance. We fixed the BLAS because some users - who just want local results - and are less enlightened than yous were using inappropriate BLAS and then complained that they were getting poor parallel performance.
We're happy to give folks like you the freedom to choose BLAS for your own reasons, although it's unfortunate that this means inferior performance from HIPO. Please make this clear in your documentation
galabovaa commentedon Aug 27, 2026
@odow This PR should fix the apple issue.
@amontoison and @filikat I also updated mycblas to use #define ACCELERATE_NEW_LAPACK if Apple Accelerate is used at compile time. I need to still think about how to do that at runtime. I also added the extras dll to the cmake install, it was missing.
@amontoison and @cvanaret Could you possibly double check if this resolves the Uno build issues?