Invalid Intel-host ARM64-target cross-compilation flag -m128bit-long-double

[matthew-brett]

We (Numpy / Scipy) are compiling OpenBLAS binaries over at https://github.com/MacPython/openblas-libs . Our arm64 builds are failing in Github Actions - e.g. https://github.com/MacPython/openblas-libs/runs/6832414065?step:7:1321

The error is:

arm64-apple-darwin20.0.0-gfortran: error: unrecognized command-line option ‘-m128bit-long-double’

I think this is a problem in OpenBLAS cross-compilation support. The -m128bit-long-double flag arises either around https://github.com/xianyi/OpenBLAS/blob/develop/Makefile.system#L536 or around https://github.com/xianyi/OpenBLAS/blob/develop/cmake/arch.cmake#L18

I believe the problem is that the NO_EXPRECISION Makefile variable is set to 0 (yes, there is extended precision) when compiling on an Intel host, but it should be set only when compiling for an Intel target.

Is that analysis right?

Should NO_EXPRECISION be set to 0 for arm64 targets?

[carlkl]

see #768 (comment)
I guess NO_EXPRECISION should be set to 1 for ARM

[matthew-brett]

Ah - I see it's more complicated than I thought. Indeed Makefile.system sets NO_EXPRECISION to 1 for anyARCH not equal to x86 or x86_64.

https://github.com/xianyi/OpenBLAS/blob/develop/Makefile.system#L494

We are compiling with TARGET=VORTEX:

https://github.com/MacPython/openblas-libs/blob/master/travis-ci/build_steps.sh#L136

Is it possible that TARGET=VORTEX does not result in ARCH being set correctly to something other than the host architecture?

[martin-frbg]

The problem appears to be that we try to define NO_EXPRECISION on any non-x86 architecture, but we do not have a concept of "GUEST ARCH". This is masked on Linux and Windows build hosts as NO_EXPRECISION is defined there by default, and perhaps that should even be the same on Mac and BSD (?)
The immediate fix would be to add another NO_EXPRECISION =1 conditional on ifdef ARMV8, and/or adding NO_EXPRECISION=1 to your build command line.

[matthew-brett]

Thanks - that's helpful.

Do you mean running the build with something like:

make NO_EXPRECISION=1 ...

?

[martin-frbg]

Yes, that should work. (Adding the ifdef ARMV8 section to Makefile.system will also fix the issue for any arm64 target as they all define ARMV8, but obviously it will not fix the general crossbuild issue for other architectures like mips or arm(32).)

[martin-frbg]

Revisiting this, I cannot reproduce the problem when cross-compiling with the current Arm GNU Toolchain on Linux/x86_64 (but with the conditional in Makefile.system that sets NO_EXPRECISION for OS=Linux commented out).
Actually, if this line from the build log

2022-06-21T14:00:28.9220360Z + make BUFFERSIZE=20 DYNAMIC_ARCH=1 USE_OPENMP=0 NUM_THREADS=64 BINARY=64 INTERFACE64=1 SYMBOLSUFFIX=64_ OBJCONV=/Users/runner/work/openblas-libs/openblas-libs/objconv/objconv TARGET=VORTEX NO_EXPRECISION=1

reflects your current commandline, you are not cross-compiling at all - I see you are setting FC=/opt/gfortran-darwin-arm64/bin/arm64-apple-darwin20.0.0-gfortran earlier but I see no corresponding CC= pointing at the C crosscompiler (and no HOSTCC to specify what to use for building host-side compile helpers like getarch). This is bound to result in the build picking the host compiler and cooking up a weird mix of x86_64 C and arm64 Fortran objects. So even with the NO_EXPRECISION now added by hand, the linker will refuse to create a library.

[matthew-brett]

Aha - I think the intention was to set the cross-compilation by setting CFLAGS to -arch arm64: https://github.com/MacPython/openblas-libs/runs/6832414065?step:7:1321#step:7:384

Were we wrong to expect that to work?

I guess that makes it hard to configure HOSTCC correctly?

[martin-frbg]

Uh, is there such a beast as a multiarch gcc on Mac where you just set -arch to a completely different architecture and it automagically builds for that ? Methinks you need a full toolchain tailored for the cross-compiling target...
HOSTCC would be as simple as HOSTCC=gcc (or clang, ...)

[matthew-brett]

I think I found our immediate problem : https://github.com/MacPython/openblas-libs/pull/84/files#diff-78c6a14663ed9f439026e56f28d155dbe6a6b7eefa152fade32c4988afbe7dbaR166

We had accidentally wiped out our carefully set CFLAGS, thus losing the arguments that specify the architecture for the C compilation.

So it does look like we can get away with just setting the -arch arguments to the compiler - but if things get more complicated, we may need to think about using a full cross-compiler rather than the default Clang with -arch arguments.

[martin-frbg]

Very interesting, did not know that LLVM can do that (and on the contrary, remember reading that it did not support cross-compilation - but that may have been specifically about flang)