Cross Compile for ARMV8

[Derrekito]

I'm trying to cross compile for ARMV8. My build parameters and errors are as followed:

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=CORTEXA73 USE_OPENMP=1

../../libopenblas_cortexa73p-r0.3.23.dev.a
make[2]: ../../libopenblas_cortexa73p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:626: ../../libopenblas_cortexa73p-r0.3.23.dev.a] Error 127
rm la_xisnan.mod
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/SRC'
make[1]: *** [Makefile:27: lapacklib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib'
make: *** [Makefile:260: netlib] Error 2

OR

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=1

../../../libopenblas_armv8p-r0.3.23.dev.a
make[2]: ../../../libopenblas_armv8p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:84: ../../../libopenblas_armv8p-r0.3.23.dev.a] Error 127
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/TESTING/MATGEN'
make[1]: *** [Makefile:49: tmglib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib'
make: *** [Makefile:261: netlib] Error 2

[martin-frbg]

the "permission denied" suggests you do not have write access in your OpenBLAS source directory. Did you perhaps unpack the source archive as root ?

[Derrekito]

I simply cloned the repo and tried compiling it using the parameters I mentioned. Also, I am able to natively compile OpenMP using just make. To be clear, I recreated the error (as a normal user) using the following steps:

rm -rf OpenBLAS
git clone https://github.com/xianyi/OpenBLAS.git
make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=

../../libopenblas_armv8p-r0.3.23.dev.a
make[2]: ../../libopenblas_armv8p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:626: ../../libopenblas_armv8p-r0.3.23.dev.a] Error 127
rm la_xisnan.mod
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/SRC'
make[1]: *** [Makefile:27: lapacklib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLASls -la | grep libopenblas_armv8p
lrwxrwxrwx 1 derrekito derrekito 32 Apr 17 15:33 libopenblas.a -> libopenblas_armv8p-r0.3.23.dev.a
-rw-rw-r-- 1 derrekito derrekito 18438972 Apr 17 15:33 libopenblas_armv8p-r0.3.23.dev.a/lapack-netlib'
make: *** [Makefile:260: netlib] Error 2

In case this helps:

ls -la | grep libopenblas_armv8p
lrwxrwxrwx  1 derrekito derrekito       32 Apr 17 15:33 libopenblas.a -> libopenblas_armv8p-r0.3.23.dev.a
-rw-rw-r--  1 derrekito derrekito 18438972 Apr 17 15:33 libopenblas_armv8p-r0.3.23.dev.a

[martin-frbg]

Strange, I cannot reproduce this (without doing something evil, as follows...) . The error message looks a bit like it is trying to invoke libopenblas.a at this point, instead of running a command (which would be either ar or ranlib) on it. What do you have in the generated OpenBLAS/lapack-netlib/make.inc file ? (An empty entry on the RANLIB= line would probably cause this, but I think this could only happen if you are actively setting RANLIB empty somewhere in your environment)

[Derrekito]

I'm also able to reproduce the error in an Ubuntu 20.04 Docker environment using my exact make line as shown earlier.

Here is my make.inc file

FC          = aarch64-linux-gnu-gfortran-11
override FFLAGS      = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -fno-tree-vectorize
FFLAGS_DRV  = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a
POPTS       = -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -pg -fno-inline
FFLAGS_NOOPT       = -O0 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a
PNOOPT      = -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -pg -fno-inline -O0
LDFLAGS     = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -lm -lpthread -lgfortran
CC          = aarch64-linux-gnu-gcc-11
override CFLAGS      = -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=20 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION="0.3.23.dev" -march=armv8-a -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME="_" -DCHAR_CNAME="" -DNO_AFFINITY -I. -DHAVE_LAPACK_CONFIG_H
AR          = ar
ARFLAGS     =  -ru
LAPACKLIB   = ../../libopenblas_armv8p-r0.3.23.dev.a
TMGLIB      = ../../../libopenblas_armv8p-r0.3.23.dev.a
BLASLIB     = ../../../libopenblas_armv8p-r0.3.23.dev.a
LAPACKELIB  = ../../../libopenblas_armv8p-r0.3.23.dev.a
LAPACKLIB_P = ../libopenblas_armv8p-r0.3.23.dev_p.a
SUFFIX      = o
PSUFFIX     = po
CEXTRALIB   = -lm -lpthread -lgfortran
TIMER       = INT_ETIME
LOADER      = aarch64-linux-gnu-gfortran-11 -pthread
BUILD_DEPRECATED      = 1
BUILD_SINGLE      = 1
BUILD_DOUBLE      = 1
BUILD_COMPLEX      = 1
BUILD_COMPLEX16      = 1
LAPACKE_WITH_TMG      = 1
SHELL = /bin/sh
PLAT = _LINUX
DRVOPTS  = $(NOOPT)
#ARCHFLAGS= $(ARFLAGS) -ru
#RANLIB   = ranlib

Changing my build arguments to this fixed the issue!

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=1 RANLIB=ranlib

I'll leave this ticket open for a short while to allow any additional remarks.

[martin-frbg]

Thanks, that appears to confirm my idea - but it is still unclear to me why (and where) you would have an empty RANLIB= entry in your environment. Would that be a default in Ubuntu 20 - but I do not think that would make sense ?

[Derrekito]

Well, the main system I tried on is Pop-OS 22.04. The Docker image is Ubuntu 20.04. The behavior was the same either way. I haven't tested on any other environment (e.g., Arch or Slackware). I believe I can share my Dockerfile if there is any additional interest in tracking down the issue. Otherwise, I'm happy to close this ticket. Please advise. I appreciate all your help!

[martin-frbg]

Can you check what running set to dump the environment tells about the presence of a RANLIB variable ? (Btw it is strange that the make.inc you posted does not have an (uncommented) RANLIB line at all - is this a hand-edited version already ? make.inc gets generated by the toplevel Makefile, and that uses unconditional echo commands to write individual lines. There is no mechanism for querying what RANLIB should be, just a simple "set it to "ranlib" (with the cross-compiler prefix if necessary) unless it is already set" in Makefile.system

[Derrekito]

I have not hand-edited any part of the OpenMP project. My only input is manually setting environment vars at the command line, appended to make, as originally shown. Anyway, running the following does not print any output:

set | grep RANLIB

As an aside, I can compile natively just fine without these issues by simply executing make without setting any additional environment variables.

[martin-frbg]

Thanks, I guess it will have to remain a mystery then - anyway glad that we could solve the problem.

[Dirreke]

I have the same Issue when cross compile for armv8 on Ubuntu22.04