Crosscompile for 64 Bit Debian 10

[visglz]

Hi Abhishek,

i am trying to crosscompile a small example libusb example for Raspberry PI OS 64 Bit (Debian 10 / Buster).

The example is basically [testlibusb.c][https://github.com/libusb/libusb/blob/v1.0.22/examples/testlibusb.c] from libusb version 1.0.22 (which is also installed in Raspberry PI OS Buster).

To crosscompile I downloaded "Raspberry Pi GCC 64-Bit Cross-Compiler Toolchains" for gcc 8.3.0, link

Besides other libs I need to fetch the aarch64 libusb from raspberry PI to a local folder, here "rootfs_part".

The problem I run into is that the libusb from Debian Buster is linked to glibc 2.28, the version Buster is build with, but the toolchain I downloaded is built with 2.24:

/opt/sdk/cross-pi-gcc-8.3.0-64/bin/aarch64-linux-gnu-gcc   -rdynamic CMakeFiles/testlibusb.dir/testlibusb.c.o  -o testlibusb  -L/home/visglz/src/vistrrdedock10/raspi/cmake/../build/raspi_libs/rootfs_part/usr/lib/aarch64-linux-gnu -lusb-1.0 -L/home/visglz/src/vistrrdedock10/raspi/cmake/../build/raspi_libs/rootfs_part/lib/aarch64-linux-gnu -ludev 
/home/visglz/src/vistrrdedock10/raspi/cmake/../build/raspi_libs/rootfs_part/lib/aarch64-linux-gnu/libudev.so: undefined reference to `getrandom@GLIBC_2.25'
/home/visglz/src/vistrrdedock10/raspi/cmake/../build/raspi_libs/rootfs_part/lib/aarch64-linux-gnu/libudev.so: undefined reference to `fcntl64@GLIBC_2.28'
/home/visglz/src/vistrrdedock10/raspi/cmake/../build/raspi_libs/rootfs_part/lib/aarch64-linux-gnu/libudev.so: undefined reference to `reallocarray@GLIBC_2.26'

$ find /opt/sdk/cross-pi-gcc-8.3.0-64 -name "libc-2*so"
/opt/sdk/cross-pi-gcc-8.3.0-64/aarch64-linux-gnu/libc/lib64/libc-2.24.so

Wouldn't it make sense to also provide a 64 Bit version with the Compiler and glibc matching for Debian Buster? I can't find such a combination in builder_64.yml currently.

Locally I built a toolchain with the following settings:

./RTBuilder_64b -g 8.3.0 -t 2

With the resulting crosscompiler I was able to crossbuild testlibusb.c and let run the resulting binary on the pi.

BTW: Great work you are doing with this project. 👍

Regards
Georg

[abhiTronix]

To crosscompile I downloaded "Raspberry Pi GCC 64-Bit Cross-Compiler Toolchains" for gcc 8.3.0, link

Besides other libs I need to fetch the aarch64 libusb from raspberry PI to a local folder, here "rootfs_part".

The problem I run into is that the libusb from Debian Buster is linked to glibc 2.28, the version Buster is build with, but the toolchain I downloaded is built with 2.24:

@visglz Thanks. Yeah, I wasn't aware buster 64bit OS is already out. I've been out of the loop lately 😄 . I'll add this to TODO list and will work on it few days.

Wouldn't it make sense to also provide a 64 Bit version with the Compiler and glibc matching for Debian Buster? I can't find such a combination in builder_64.yml currently.

Locally I built a toolchain with the following settings:

./RTBuilder_64b -g 8.3.0 -t 2
With the resulting crosscompiler I was able to crossbuild testlibusb.c and let run the resulting binary on the pi.

Kudos for finding this 🎉 . I've already predicted that Raspbian will soon release 64 bit version so I kept this hidden option for it. Also, Thanks again for testing it.

[visglz]

Thanks for the quick answer. 👍

I was thinking about submitting a PR already, but I made a rather strange observation. Maybe you can have a look at it?

I am trying to compile and link a test program which uses another external library from raspberry pi.

In my case I choose pngtest.c. pngtest.c is also installed with the libpng-dev package.

Besides pngtest.c there is a include and lib folder with the files from the raspi rootfs:

.
+-- include
¦   +-- pngconf.h
¦   +-- png.h
¦   +-- pnglibconf.h
¦   +-- zconf.h
¦   +-- zlib.h
+-- lib
¦   +-- libpng16.a
¦   +-- libpng16.so -> libpng16.so.16.36.0
¦   +-- libpng16.so.16 -> libpng16.so.16.36.0
¦   +-- libpng16.so.16.36.0
¦   +-- libpng.so -> libpng16.so
¦   +-- libz.so -> libz.so.1
¦   +-- libz.so.1 -> libz.so.1.2.11
¦   +-- libz.so.1.2.11
+-- pngtest.c

To quick check I build first with my local compiler (the last line in pngtest.c with the header version check needs to be commented out).

gcc pngtest.c -lpng

-> Works, a.out is generated and can be run on the Ubuntu Workstation.

Then I build with prebuild compiler I downloaded previously link:

/opt/sdk/cross-pi-gcc-8.3.0-64/bin/aarch64-linux-gnu-gcc -Llib -Iinclude -lpng -lz pngtest.c

-> Works, a.out is generated and can be run on raspberry pi.

Then with the same commandline I try to build with my locally build compiler (note the appended _2 in the path):

$ /opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/aarch64-linux-gnu-gcc -Llib -Iinclude -lpng -lz pngtest.c
pngtest.c: In function 'test_one_file':
pngtest.c:1607: warning: "pass_height" redefined
 #        define pass_height height
 
pngtest.c:1544: note: this is the location of the previous definition
 #              define pass_height frame_height
 
$ /opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/../../../../aarch64-linux-gnu/bin/ld: warning: libm.so.6, needed by lib/libpng.so, not found (try using -rpath or -rpath-link)
/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/../../../../aarch64-linux-gnu/bin/ld: lib/libpng.so: undefined reference to `pow@GLIBC_2.17'
collect2: error: ld returned 1 exit status

But now the linker does not find libm anymore. Here is the relevant strace from the compiler run:

openat(AT_FDCWD, "/aarch64-linux-gnu/lib64/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/aarch64-linux-gnu/lib/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/aarch64-linux-gnu/lib64/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/aarch64-linux-gnu/lib/libm.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)

The math library of the standard lib is a dependency of libpng. I guess the linker parses libm.so.6 out of libpng.sos ELF header.

Interestingly when I add "-lm" the compiler finds the libm:

$ /opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/aarch64-linux-gnu-gcc -Llib -Iinclude -lpng -lz -lm pngtest.c

Here is the strace with "-lm" appended:

openat(AT_FDCWD, "lib/libm.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "lib/libm.a", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/libm.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/libm.a", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/libm.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/libm.a", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/../../../../aarch64-linux-gnu/lib/../lib64/libm.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../lib/gcc/aarch64-linux-gnu/8.3.0/../../../../aarch64-linux-gnu/lib/../lib64/libm.a", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../aarch64-linux-gnu/libc/lib/../lib64/libm.so", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../aarch64-linux-gnu/libc/lib/../lib64/libm.a", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../aarch64-linux-gnu/libc/usr/lib/../lib64/libm.so", O_RDONLY) = 9

Do you know why with my local build compiler wants to have the "-lm" appended in contrast to the prebuild version?

I am also a bit curious if I build the compiler/toolchain correctly. First I run the following command:

BUILDDIR=$PWD/build ./RTBuilder_64b -g 8.3.0 -t 2

But then I realized that github CI uses other script in the CI subfolder.

Do those scripts use maybe another configuration?

[abhiTronix]

/opt/sdk/cross-pi-gcc-8.3.0-64/bin/aarch64-linux-gnu-gcc -Llib -Iinclude -lpng -lz pngtest.c

-> Works, a.out is generated and can be run on raspberry pi.

@visglz What's the strace on this command (i.e. with downloaded project's binary)? I'm guessing your locally build binaries are incomplete, but not sure.

[abhiTronix]

@visglz Also can you run nm command on both a.out (i.e. from project's GCC binary and your locally build) and paste the output here.

[visglz]

/opt/sdk/cross-pi-gcc-8.3.0-64/bin/aarch64-linux-gnu-gcc -Llib -Iinclude -lpng -lz pngtest.c
-> Works, a.out is generated and can be run on raspberry pi.

@visglz What's the strace on this command (i.e. with downloaded project's binary)? I'm guessing your locally build binaries are incomplete, but not sure.

Please see here for the relevant strace section:

openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64/bin/../aarch64-linux-gnu/libc/etc/ld.so.conf", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64/bin/../aarch64-linux-gnu/libc/etc/ld.so.conf", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64/bin/../aarch64-linux-gnu/libc/aarch64-linux-gnu/lib64/libm.so.6", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64/bin/../aarch64-linux-gnu/libc/usr/local/lib64/libm.so.6", O_RDONLY) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
openat(AT_FDCWD, "/opt/sdk/cross-pi-gcc-8.3.0-64/bin/../aarch64-linux-gnu/libc/lib64/libm.so.6", O_RDONLY) = 20

My guess is also that my local build gcc is somewhat different :-) , so I re-run ./RTBuilder_64b two times - same result.

Interestingly the compiler finds the filename without problems when asked:

$ /opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/aarch64-linux-gnu-gcc -print-file-name=libm.so.6
/opt/sdk/cross-pi-gcc-8.3.0-64_2/bin/../aarch64-linux-gnu/libc/lib/../lib64/libm.so.6

Please find attached the output of nm:

• nm_prebuild_gcc.txt
• nm_ubuntu_gcc.txt

For the locally build gcc I also dumped the specs (gcc --dumpspecs) and printed a few variables like compiled in sysroots, see the following attachments:

• gcc_dump_specs.txt
• gcc_print_variables.txt

[abhiTronix]

@visglz Thanks. There were some important sysroot patches missing from build-scripts, and is fixed in commit fa53861. Can you try that branch by re-cloning as follows:

git clone https://github.com/abhiTronix/raspberry-pi-cross-compilers.git && cd raspberry-pi-cross-compilers
git checkout patch@buildscript

And then rebuild your local binaries by running ./RTBuilder_64b again.