Getting a Illegal instruction segfault on arm (Raspberry Pi 1)

[weiznich]

Originally reported as tokio-rs/tokio-core#324

I'm trying to run some future related code on an old raspberry pi 1. After cross compiling a binary construction a reactor and spawning a future on it the program segfaults with an illegal hardware instruction error.
Programs without a reactor are running fine.

Example code:

extern crate tokio_core;
extern crate futures;

fn main() {
   let core = tokio_core::reactor::Core::new().unwrap();
       core.handle().spawn(futures::future::ok(()));
    println!("Hello, world!");
}

Versions:

• Rustc: rustc 1.25.0 (84203cac6 2018-03-25), rustc 1.27.0-nightly (79252ff4e 2018-04-29)
• tokio_core: 0.1.17
• futures: 0.1.21
• target: arm-unknown-linux-musleabihf

[estebank]

@weiznich could you verify if this ever worked on previous versions of the compiler?

[mattico]

Could you collect a stacktrace? Something like this should work:

$ ulimit -c unlimited # enable core dumps
$ cargo run
$ gdb ./target/debug/PROGRAM core.PROGRAM.PID
(gdb) bt

[weiznich]

Could you collect a stacktrace?

(I've just run the test program in gdb)

(gdb) bt
#0  printf_core (f=f@entry=0x0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=ap@entry=0x7effe6ec, nl_arg=nl_arg@entry=0x7effe718, nl_type=nl_type@entry=0x7effe6f0) at src/stdio/vfprintf.c:618
#1  0x0010179c in vfprintf (f=f@entry=0x7effe7f0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=ap@entry=...) at src/stdio/vfprintf.c:666
#2  0x001019e8 in vsnprintf (s=s@entry=0x0, n=n@entry=0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=...) at src/stdio/vsnprintf.c:54
#3  0x00105894 in __dl_vseterr (fmt=0x11a1d1 "Symbol not found: %s", fmt@entry=0x7efff55c "<\365\377~\354\365\377~\344\365\377~\001", ap=..., ap@entry=...) at src/ldso/dlerror.c:34
#4  0x001058ec in __dl_seterr (fmt=0x11a1d1 "Symbol not found: %s") at src/ldso/dlerror.c:51
#5  0x0010581c in stub_dlsym (p=<optimized out>, s=<optimized out>, ra=<optimized out>) at src/ldso/__dlsym.c:9
#6  0x000c8dc4 in mio::sys::unix::dlsym::fetch (name=...) at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/dlsym.rs:43
#7  0x000c8b60 in <mio::sys::unix::dlsym::DlSym<F>>::get (self=0x14506c <mio::sys::unix::epoll::Selector::new::epoll_create1>) at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/dlsym.rs:30
#8  0x000c8e7c in mio::sys::unix::epoll::Selector::new () at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/epoll.rs:38
#9  0x000bd148 in mio::poll::Poll::new () at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.14/src/poll.rs:655
#10 0x000a934c in tokio_reactor::Reactor::new () at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.1/src/lib.rs:215
#11 0x00053b78 in tokio::runtime::builder::Builder::build (self=0x7effeff0) at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.5/src/runtime/builder.rs:94
#12 0x00058ccc in tokio::runtime::Runtime::new () at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.5/src/runtime/mod.rs:260
#13 0x0004c72c in tokio_core::reactor::Core::new () at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:129
#14 0x000129a8 in tokio_segfault_test::main () at src/main.rs:5

could you verify if this ever worked on previous versions of the compiler?

The oldest version that compiles the test snipped is rust 1.21. (Due to some features in crossbeam). It is already broken there.

---

Compiling the same code using the arm-unknown-linux-gnueabihf target produces a segfault due to a illegal instruction.

(gdb) bt
#0  0x00408602 in <crossbeam_utils::cache_padded::CachePadded<T> as core::ops::deref::DerefMut>::deref_mut (
    self=0x7efff5b4)
    at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.3.2/src/cache_padded.rs:147
Backtrace stopped: Cannot access memory at address 0x4

Using the arm-unknown-linux-gnueabi target produces a segmentation fault due to some memory issue 😟

(gdb) bt
#0  0x76ddfdd8 in _dl_update_slotinfo () from /usr/lib/ld-linux.so.3
#1  0x00000160 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Using the arm-unknown-linux-musleabi target produces a binary that runs without segfault.

[mattico]

Some raspi distros use the soft-float ABI, apparently, which could explain the gnueabihf issue.

Not sure about the second issue, maybe a glibc versioning issue? Can you try compiling on the raspi?

[weiznich]

Some raspi distros use the soft-float ABI, apparently , which could explain the gnueabihf issue.

According to that link Arch Linux (the distro running on the Pi) supports the *hf target since 2012. Also a program without a tokio reactor works just fine. I assume this won't be the case if I've chosen the wrong target?

Can you try compiling on the raspi?

I will try that later.

[weiznich]

Can you try compiling on the raspi?

Building the code above on a pi using the arm-unknown-linux-gnueabihf target produces a working binary.

[parched]

Looks like the illegal instruction is in musl, where did you get musl from, maybe it's build for Armv7? Can you print the actual instruction in gdb that causes the problem?

[weiznich]

Looks like the illegal instruction is in musl, where did you get musl from, maybe it's build for Armv7?

All targets where installed via rustup. For building against musl I used this docker file as base (I've used the arm-musleabihf variant.)

Can you print the actual instruction in gdb that causes the problem?

   │0xffff8 <printf_core>           push   {r4, r5, r6, r7, r8, r9, r10, r11, lr}
   │0xffffc <printf_core+4>         vpush  {d8-d14} 
  >│0x100000 <printf_core+8>        vmov.f64       d10, #112       ; 0x3f800000  1.0    
   │0x100004 <printf_core+12>       vmov.f64       d11, #120       ; 0x3fc00000  1.5      
   │0x100008 <printf_core+16>       vldr   d8, [pc, #904]  ; 0x100398 <printf_core+928>   
   │0x10000c <printf_core+20>       vldr   d9, [pc, #908]  ; 0x1003a0 <printf_core+936>

[parched]

That's a VFPv3 instruction (raspi 1 has VFPv2), so it looks like musl has been compiled with the wrong flags.

[weiznich]

That's a VFPv3 instruction (raspi 1 has VFPv2), so it looks like musl has been compiled with the wrong flags.

So I've tried to rebuild the whole musl toolchain from scratch using the same flags as the gcc installed on the raspberry pi, but I'm getting the same Illegal Instruction error as before.

gcc on pi:

$ gcc -v
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/armv6l-unknown-linux-gnueabihf/7.3.1/lto-wrapper
Target: armv6l-unknown-linux-gnueabihf
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://github.com/archlinuxarm/PKGBUILDs/issues --enable-languages=c,c++,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --disable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --host=armv6l-unknown-linux-gnueabihf --build=armv6l-unknown-linux-gnueabihf --with-arch=armv6 --with-float=hard --with-fpu=vfp
Thread model: posix
gcc version 7.3.1 20180406 (GCC)

(musl)gcc in docker container

$ gcc -v
Using built-in specs.
COLLECT_GCC=arm-unknown-linux-musleabihf-gcc
COLLECT_LTO_WRAPPER=/usr/local/musl/bin/../libexec/gcc/arm-unknown-linux-musleabihf/7.2.0/lto-wrapper
Target: arm-unknown-linux-musleabihf
Configured with: ../src_gcc/configure --enable-languages=c,c++ --with-float=hard CFLAGS='-g0 -Os' CXXFLAGS='-g0 -Os' LDFLAGS=-s --disable-nls --with-debug-prefix-map=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf= --enable-languages=c,c++ --disable-libquadmath --disable-decimal-float --disable-multilib --with-arch=armv6 --with-float=hard --with-fpu=vfp --disable-werror --target=arm-unknown-linux-musleabihf --prefix= --libdir=/lib --disable-multilib --with-sysroot=/arm-unknown-linux-musleabihf --enable-tls --disable-libmudflap --disable-libsanitizer --disable-gnu-indirect-function --disable-libmpx --enable-deterministic-archives --enable-libstdcxx-time --with-build-sysroot=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_sysroot AR_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/ar AS_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/gas/as-new LD_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/ld/ld-new NM_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/nm-new OBJCOPY_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/objcopy OBJDUMP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/objdump RANLIB_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/ranlib READELF_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/readelf STRIP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/strip-new
Thread model: posix
gcc version 7.2.0 (GCC

I'm not sure if I missed something but I thing I set all relevant flags to be the same (--with-arch=armv6 --with-float=hard --with-fpu=vfp )

[mattico]

Did you also recompile musl libc with that compiler?

[weiznich]

I think so. The docker file uses this to build the musl cross compiler. I've build everything from scratch.

[mattico]

Wow this is a real cross-compile hell.

Rustc might be using a different gcc. Try cargo rustc --target=arm-unknown-linux-musleabihf -- -C linker=gcc.

Maybe rustc/gcc is finding libc in the wrong place? We can find out by looking at the output of ld --verbose. There's probably an easier way to do this but this is what I did:

# note that I'm using gnueabihf on ubuntu so the commands are a bit different
$ cargo +nightly -v rustc --target=arm-unknown-linux-gnueabihf -- -C linker=arm-linux-gnueabihf-gcc -Z print-link-args
   Compiling test-cross v0.1.0 (file:///home/matt/test-cross)
     Running `rustc --crate-name test_cross <snip>
# Paste the rustc invocation here so we get the print-link-args output
$ rustc --crate-name test_cross  <snip>
"arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" <snip>
# paste the gcc invocation here, and add `-v -Wl,--verbose`
$ "arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" <snip> -v -Wl,--verbose
<lots of output>

In the ld --verbose output it should say where it's finding libc.a. If that's not the libc.a from your musl toolchain, then that's your problem.