aarch64 musl binaries panic since 2018-02-05 nightly

[tjkirch]

aarch64-unknown-linux-musl binaries crash immediately when built using Rust nightly since 2018-02-05, including the current beta, 1.25.0-beta.9.

This happens in debug and release builds.

It works fine with 2018-02-04, or with stable Rust 1.24.1. This is building on an x86_64-unknown-linux-gnu host, which shows no errors or warnings, and running on an embedded Linux 4.9 device.

I tried this code:

A fresh cargo new --bin testme. (Same thing with a completely empty "main {}" function.)

I expected to see this happen:

The binary should run without panicking. With nightly 2018-02-04, it looks like this, through strace:

execve("/tmp/testme", ["/tmp/testme"], 0x7fe28422a0 /* 9 vars */) = 0
mmap(NULL, 448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f87569000
set_tid_address(0x7f87569038)           = 19524
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x425660}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x408844, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO|SA_ONSTACK, sa_restorer=0x425660}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=0x408844, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO|SA_ONSTACK, sa_restorer=0x425660}, NULL, 8) = 0
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f87566000
Hello, world!
sigaltstack({ss_sp=0x7f87566000, ss_flags=0, ss_size=12288}, NULL) = 0
brk(NULL)                               = 0x451000
brk(0x452000)                           = 0x452000
write(1, "Hello, world!\n", 14)         = 14
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=12288}, NULL) = 0
munmap(0x7f87566000, 12288)             = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Instead, with 2018-02-05+, this happened:

thread panicked while processing panic. aborting.
Trace/breakpoint trap (core dumped)

With strace:

execve("/tmp/testme", ["/tmp/testme"], 0x7fe4522920 /* 9 vars */) = 0
mmap(NULL, 520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82fb0000
set_tid_address(0x7f82fb0040)           = 6644
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x424780}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x40cad8, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO|SA_ONSTACK, sa_restorer=0x424780}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=0x40cad8, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO|SA_ONSTACK, sa_restorer=0x424780}, NULL, 8) = 0
sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82fad000
sigaltstack({ss_sp=0x7f82fad000, ss_flags=0, ss_size=12288}, NULL) = 0
brk(NULL)                               = 0x44f000
brk(0x450000)                           = 0x450000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
write(2, "thread panicked while processing panic. aborting.\n", 50thread panicked while processing panic. aborting.
) = 50
--- SIGTRAP {si_signo=SIGTRAP, si_code=TRAP_BRKPT, si_pid=4242620, si_uid=0} ---
+++ killed by SIGTRAP (core dumped) +++
Trace/breakpoint trap (core dumped)

Here's the backtrace from the core, using gdb 7.1.2:

(gdb) bt
#0  std::panicking::rust_panic_with_hook::h32ba6c175ffd549b () at libstd/panicking.rs:562
#1  0x000000000040bbb4 in std::panicking::begin_panic::h54415e6a8d568874 () at libstd/panicking.rs:537
#2  0x000000000040bb24 in std::panicking::begin_panic_fmt::he7ac75f1ed16a44d () at libstd/panicking.rs:521
#3  0x000000000040bab8 in rust_begin_unwind () at libstd/panicking.rs:497
#4  0x0000000000425740 in core::panicking::panic_fmt::he44a5deca1206c59 () at libcore/panicking.rs:71
#5  0x0000000000401e34 in core::result::unwrap_failed::h3872e2fc78d2079d () at /checkout/src/libcore/macros.rs:23
#6  0x000000000040316c in _$LT$core..result..Result$LT$T$C$$u20$E$GT$$GT$::expect::hd60da7509cc07397 () at /checkout/src/libcore/result.rs:809
#7  _$LT$core..cell..RefCell$LT$T$GT$$GT$::borrow::hb3e1337e98c3fd8c () at /checkout/src/libcore/cell.rs:692
#8  std::sys_common::thread_info::ThreadInfo::with::_$u7b$$u7b$closure$u7d$$u7d$::h8953affb3be93291 () at libstd/sys_common/thread_info.rs:26
#9  _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h26bdeb702705cd55 () at libstd/thread/local.rs:377
#10 0x000000000040b3d0 in std::sys_common::thread_info::ThreadInfo::with::hea74330e81f8e238 () at libstd/sys_common/thread_info.rs:25
#11 std::sys_common::thread_info::current_thread::h1ca059562bf90a53 () at libstd/sys_common/thread_info.rs:38
#12 std::panicking::default_hook::h1c4df4ccc9dbcc8f () at libstd/panicking.rs:366
#13 0x000000000040bd5c in std::panicking::rust_panic_with_hook::h32ba6c175ffd549b () at libstd/panicking.rs:576
#14 0x000000000040bbb4 in std::panicking::begin_panic::h54415e6a8d568874 () at libstd/panicking.rs:537
#15 0x000000000040bb24 in std::panicking::begin_panic_fmt::he7ac75f1ed16a44d () at libstd/panicking.rs:521
#16 0x000000000040bab8 in rust_begin_unwind () at libstd/panicking.rs:497
#17 0x0000000000425740 in core::panicking::panic_fmt::he44a5deca1206c59 () at libcore/panicking.rs:71
#18 0x0000000000401e34 in core::result::unwrap_failed::h3872e2fc78d2079d () at /checkout/src/libcore/macros.rs:23
#19 0x000000000040cf80 in _$LT$core..result..Result$LT$T$C$$u20$E$GT$$GT$::expect::hd60da7509cc07397 () at /checkout/src/libcore/result.rs:809
#20 _$LT$core..cell..RefCell$LT$T$GT$$GT$::borrow::hb3e1337e98c3fd8c () at /checkout/src/libcore/cell.rs:692
#21 std::sys_common::thread_info::ThreadInfo::with::_$u7b$$u7b$closure$u7d$$u7d$::h9b784f47e02471dc () at libstd/sys_common/thread_info.rs:26
#22 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h679d37f9412a06de () at libstd/thread/local.rs:377
#23 std::sys_common::thread_info::ThreadInfo::with::hbe63383438d236c7 () at libstd/sys_common/thread_info.rs:25
#24 std::sys_common::thread_info::stack_guard::h9e3b15191b000f90 () at libstd/sys_common/thread_info.rs:42
#25 std::sys::unix::stack_overflow::imp::signal_handler::h7311ce70e0648552 () at libstd/sys/unix/stack_overflow.rs:105
#26 <signal handler called>
#27 core::sync::atomic::atomic_add::h28be0c5b8884ec73 () at /checkout/src/libcore/sync/atomic.rs:1529
#28 core::sync::atomic::AtomicUsize::fetch_add::h6d05bf1ec9fefec2 () at /checkout/src/libcore/sync/atomic.rs:1285
#29 _$LT$alloc..arc..Arc$LT$T$GT$$u20$as$u20$core..clone..Clone$GT$::clone::h6010f845e6b4f7cd () at /checkout/src/liballoc/arc.rs:713
#30 _$LT$std..thread..Thread$u20$as$u20$core..clone..Clone$GT$::clone::h676320869dc42c9e () at libstd/thread/mod.rs:995
#31 std::sys_common::thread_info::current_thread::_$u7b$$u7b$closure$u7d$$u7d$::h9421a40bb49e2782 () at libstd/sys_common/thread_info.rs:38
#32 std::sys_common::thread_info::ThreadInfo::with::_$u7b$$u7b$closure$u7d$$u7d$::h8953affb3be93291 () at libstd/sys_common/thread_info.rs:32
#33 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::h26bdeb702705cd55 () at libstd/thread/local.rs:377
#34 0x000000000040b3d0 in std::sys_common::thread_info::ThreadInfo::with::hea74330e81f8e238 () at libstd/sys_common/thread_info.rs:25
#35 std::sys_common::thread_info::current_thread::h1ca059562bf90a53 () at libstd/sys_common/thread_info.rs:38
#36 std::panicking::default_hook::h1c4df4ccc9dbcc8f () at libstd/panicking.rs:366
#37 0x000000000040bd5c in std::panicking::rust_panic_with_hook::h32ba6c175ffd549b () at libstd/panicking.rs:576
#38 0x000000000040bc1c in std::panicking::begin_panic::hd820cd84b7494cee () at libstd/panicking.rs:537
#39 0x000000000040a854 in std::sys_common::thread_info::set::_$u7b$$u7b$closure$u7d$$u7d$::he4d3a0675b98919e () at libstd/sys_common/thread_info.rs:46
#40 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::ha14bf0d7b1a87bd1 () at libstd/thread/local.rs:377
#41 _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::he00cc2566a9cc88e () at libstd/thread/local.rs:288
#42 std::sys_common::thread_info::set::he3dc668f6080c88a () at libstd/sys_common/thread_info.rs:46
#43 0x000000000040bfa4 in std::rt::lang_start_internal::h60dd5b329f127bc1 () at libstd/rt.rs:51
#44 0x0000000000400304 in main ()

Meta

rustc --version --verbose:

Any nightly Rust since 2018-02-05, installed through rustup. Example:

rustc 1.25.0-nightly (0c6091fbd 2018-02-04)
binary: rustc
commit-hash: 0c6091fbd0eee290c651f73be899f221eeab3c05
commit-date: 2018-02-04
host: x86_64-unknown-linux-gnu
release: 1.25.0-nightly
LLVM version: 4.0

My ~/.cargo/config:

[target.aarch64-unknown-linux-musl]
linker = "aarch64-unknown-linux-musl-gcc"
rustflags = [
  "-C", "link-arg=-lgcc",
  "-C", "target-feature=+crt-static"
]

aarch64-unknown-linux-musl-gcc is from GCC 7.2.0 via Buildroot 2017.08.

Note: I've tried adding -C llvm-args=-fast-isel per #48673 but it made no difference.

[tjkirch]

I see the libc repo had a similar-sounding issue, but it was back in November 2017, and Rust builds have been working for me until February 5 2018. It may have prevented more visibility, though. See: rust-lang/libc#856 and rust-lang/libc@bea4879eec9a1

[tjkirch]

A colleague found that dynamically linking libgcc (and libc) using "target-feature=-crt-static" works around this issue.

[parched]

I wonder if it's related to #46566

[nikomatsakis]

If indeed it worked in 2018-02-04 and failed in 2018-02-05, then the suspicious commit range is 3d292b7 .. 0c6091f (commits), presuming I am interpreting this rustup output correctly

> rustup install nightly-2018-02-04
info: syncing channel updates for 'nightly-2018-02-04-x86_64-unknown-linux-gnu'

  nightly-2018-02-04-x86_64-unknown-linux-gnu unchanged - rustc 1.25.0-nightly (3d292b793 2018-02-03)

> rustup install nightly-2018-02-05
info: syncing channel updates for 'nightly-2018-02-05-x86_64-unknown-linux-gnu'

  nightly-2018-02-05-x86_64-unknown-linux-gnu unchanged - rustc 1.25.0-nightly (0c6091fbd 2018-02-04)

[nikomatsakis]

@tjkirch can you validate the commits from the two nightlies? That commit range looks a bit suspicious. Just include the -vV output from each rustc that you tested with.

[nikomatsakis]

triage: P-high

We should figure out what is happening here.