Many tests failing with "Couldn't compile the test" on 32-bit PowerPC with 1.49 beta

[glaubitz]

Since 1.49 beta, many tests fail on 32-bit PowerPC with the error that the test could not be compiled:

running 1 test
test src/test/rustdoc/async-move-doctest.rs - (line 8) ... FAILED

failures:

---- src/test/rustdoc/async-move-doctest.rs - (line 8) stdout ----
Couldn't compile the test.

failures:
    src/test/rustdoc/async-move-doctest.rs - (line 8)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

I have not bisected the issue yet.

Full log in: https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=powerpc&ver=1.49.0%7Ebeta.4%2Bdfsg1-1%7Eexp1&stamp=1611482142&raw=0

CC @jrtc27 @infinity0 @smaeul @mator

[bjorn3]

I am seeing a lot of signal 4 (SIGILL) errors in the build log.

[glaubitz]

Ah, that reminds me of one bug in LLVM: https://bugs.llvm.org/show_bug.cgi?id=46683

I'll check whether this one fixes it: llvm/llvm-project@a5d161c

[glaubitz]

Debian's LLVM 11 package (llvm-toolchain-11) already contains the fix, so it can't be this.

I'm trying to bisect this now in the Rust compiler itself but I just stumbled into the problem that a cross build no longer builds a full stage2 compiler with the std compiler which can be used on the target system, so I will first have to figure out what changed in Rust here.

[bjorn3]

You can pass --stage 2 to ./x.py. The default changed to --stage 1.

[glaubitz]

You can pass --stage 2 to ./x.py. The default changed to --stage 1.

Thanks, that's what I guessed already, I just couldn't find the change in the vast amount of other changes ;).

[bjorn3]

https://blog.rust-lang.org/inside-rust/2020/08/30/changes-to-x-py-defaults.html: #73964

[glaubitz]

Just cross-compiled 1.50 and the issue is still there:

(sid_powerpc-dchroot)glaubitz@perotto:~/rustc/rustc-1.50.0+dfsg1$ CARGO_PKG_HOMEPAGE='https://github.com/alexcrichton/cfg-if' CARGO_PKG_LICENSE=MIT/Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=cfg-if CARGO_PKG_REPOSITORY='https://github.com/alexcrichton/cfg-if' CARGO_PKG_VERSION=0.1.10 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=10 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps:/home/glaubitz/tmp/usr/local/lib:' /home/glaubitz/tmp/usr/local/bin/rustc --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
{"artifact":"/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps/cfg_if-144fa4f9a0a129e5.d","emit":"dep-info"}
Illegal instruction
(sid_powerpc-dchroot)glaubitz@perotto:~/rustc/rustc-1.50.0+dfsg1$

[glaubitz]

I cannot get a sensible backtrace, unfortunately:

(gdb) r --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
Starting program: /home/glaubitz/tmp/usr/local/bin/rustc --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
[New Thread 0xede3f0f0 (LWP 256221)]
{"artifact":"/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps/cfg_if-144fa4f9a0a129e5.d","emit":"dep-info"}

Thread 2 "rustc" received signal SIGILL, Illegal instruction.
[Switching to Thread 0xede3f0f0 (LWP 256221)]
0xf4f59bbc in std::thread::local::fast::Key$LT$T$GT$::try_initialize::h23c87cba01958f05 ()
   from /home/glaubitz/tmp/usr/local/bin/../lib/librustc_driver-9cf50d6324ac9732.so
(gdb) bt
#0  0xf4f59bbc in std::thread::local::fast::Key$LT$T$GT$::try_initialize::h23c87cba01958f05 ()
   from /home/glaubitz/tmp/usr/local/bin/../lib/librustc_driver-9cf50d6324ac9732.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

[glaubitz]

I'm bisecting this now. I think it might actually be a bug in the Rust compiler and not LLVM.

[glaubitz]

git bisect has lead me to 0328e69:

0328e69287b083af4d5d4b49cfc9175e9c82c88e is the first bad commit
commit 0328e69287b083af4d5d4b49cfc9175e9c82c88e
Author: Josh Triplett <josh@joshtriplett.org>
Date:   Sun Oct 25 22:11:20 2020 -0700

    Compile tools and internal libraries with the initial-exec TLS model
    
    This should produce more efficient code, with fewer calls to
    __tls_get_addr. The tradeoff is that libraries using it won't work with
    dlopen, but that shouldn't be a problem for tools or for our own
    internal libraries.
    
    Co-authored-by: Mark Rousskov <mark.simulacrum@gmail.com>

 src/bootstrap/builder.rs | 8 ++++++++
 src/bootstrap/lib.rs     | 4 ++++
 2 files changed, 12 insertions(+)