Disabling jemalloc fails when doing a local rebuild

[aidanhs]

Edit: see #39054 (comment)

---

~/Desktop/rust/rust-tmp $ rustc --version
rustc 1.16.0-nightly (1a2ed98d3 2017-01-13)
~/Desktop/rust/rust-tmp $ git show | head -n4
commit b0c52c587fe9ba287053359fff5ed886b7edb27c
Merge: 927c55d 3a79f2e
Author: bors <bors@rust-lang.org>
Date:   Fri Jan 13 07:24:53 2017 +0000
~/Desktop/rust/rust-tmp $ ./configure --enable-local-rust --disable-jemalloc --llvm-root=/usr/lib/llvm-3.8 --disable-codegen-tests
[...]
~/Desktop/rust/rust-tmp $ ./x.py build --stage 0 -j3 -v
[...]
     Running `/home/aidanhs/Desktop/rust/rust-tmp/build/bootstrap/debug/rustc --crate-name std src/libstd/lib.rs --crate-type dylib --crate-type rlib --emit=dep-info,li
nk -C prefer-dynamic -C opt-level=2 --cfg 'feature="backtrace"' --cfg 'feature="panic-unwind"' --cfg 'feature="panic_unwind"' -C metadata=d6c751cd700544ff -C extra-file
name=-d6c751cd700544ff --out-dir /home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unk
nown-linux-gnu -L dependency=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/release/deps --extern panic_unwind=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-g
nu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_unwind-8c67638596b8d567.rlib --extern libc=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-g
nu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liblibc-3eca0ce6ddae7cf0.rlib --extern rand=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage
0-std/x86_64-unknown-linux-gnu/release/deps/librand-3a9e8dccb0763fd8.rlib --extern alloc_system=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage
0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_system-92088ad3ca26d86d.rlib --extern std_unicode=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gn
u/stage0-std/x86_64-unknown-linux-gnu/release/deps/libstd_unicode-a77984d556d80178.rlib --extern panic_abort=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-li
nux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_abort-2a59489bf9653084.rlib --extern collections=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unkn
own-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcollections-4ee10d7be4610b32.rlib --extern compiler_builtins=/home/aidanhs/Desktop/rust/rust-tmp/build
/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-47800f01f94aef26.rlib --extern alloc=/home/aidanhs/Desktop/rust/rust-tmp
/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc-3b92c602cb553647.rlib --extern core=/home/aidanhs/Desktop/rust/rust-tmp/build/
x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-d0d04b66da2d3e03.rlib --extern unwind=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64
-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libunwind-c2bfc8772a2653f6.rlib -L native=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-l
inux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/std-89e6156878b3b964/out/.libs --cfg cargobuild -l static=backtrace -l dl -l rt -l pthread -L native=/home/ai
danhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-6ba51432c6685f3f/out`
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

Seems to get past there fine if I remove --enable-local-rust.

[aidanhs]

Ok, rustup default beta-2016-12-20 works. That's the version rustbuild will download if left to its own devices. Bisecting built nightlies as my local rust gives this:

• nightly-2016-12-19 (rustc 1.15.0-nightly (71c06a56a 2016-12-18)) - works
• [no nightlies]
• nightly-2016-12-29 - unrelated errors on some i128/u128 stuff before it gets to the relevant point
• [no nightlies]
• nightly-2017-01-04 (rustc 1.16.0-nightly (468227129 2017-01-03)) - broken

[aidanhs]

Note that this is really quick to reproduce (~1min), as it happens on the initial build of stage0 std. Here's the full log from ./x.py build --stage 0 -j1:

~/Desktop/rust/rust-tmp $ ./x.py build --stage 0 -j1                                                                                                                    
   Compiling build_helper v0.1.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/build_helper)
   Compiling libc v0.2.17
   Compiling gcc v0.3.40
   Compiling rustc-serialize v0.3.19
   Compiling getopts v0.2.14
   Compiling filetime v0.1.10
   Compiling cmake v0.1.18
   Compiling num_cpus v0.2.13
   Compiling toml v0.1.30
   Compiling bootstrap v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/bootstrap)
    Finished debug [unoptimized] target(s) in 16.84 secs
Synchronising submodule url for 'src/compiler-rt'
Synchronising submodule url for 'src/jemalloc'
Synchronising submodule url for 'src/liblibc'
Synchronising submodule url for 'src/llvm'
Synchronising submodule url for 'src/rt/hoedown'
Synchronising submodule url for 'src/rust-installer'
HEAD is now at a8fc4c1 Merge pull request #28 from xen0n/preprocessor-firefighting
HEAD is now at 9858987 [WIP] sparc64-linux support
HEAD is now at a3736a0 Merge pull request #6 from intelfx/patch-1
HEAD is now at 4f99485 Merge pull request #54 from brson/docdir
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling libc v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/rustc/libc_shim)
   Compiling core v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcore)
   Compiling alloc v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/liballoc)
   Compiling build_helper v0.1.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/build_helper)
   Compiling rand v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/librand)
   Compiling gcc v0.3.40
   Compiling std v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libstd)
   Compiling panic_abort v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libpanic_abort)
   Compiling compiler_builtins v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcompiler_builtins)
   Compiling unwind v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libunwind)
   Compiling alloc_system v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/liballoc_system)
   Compiling panic_unwind v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libpanic_unwind)
   Compiling std_unicode v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libstd_unicode)
   Compiling collections v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcollections)
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

To learn more, run the command again with --verbose.


command did not execute successfully: "/home/aidanhs/.cargo/bin/cargo" "build" "-j" "1" "--target" "x86_64-unknown-linux-gnu" "--release" "--features" "panic-unwind backtrace" "--manifest-path" "/home/aidanhs/Desktop/rust/rust-tmp/src/rustc/std_shim/Cargo.toml"
expected success, got: exit code: 101

This issue may get a lot more attention once the stage0 downloaded compiler has its version bumped, as I suspect --disable-jemalloc will be totally broken.

Applying #37975 seems to fix it.

[aidanhs]

Ah I see, the more recent compiler is being detected as a local rebuild compiler, so rustbuild assumes it has a compiler with stage1 features and therefore performs a stage1 build, which seems to assume that jemalloc is available.

On a positive note, this should never affect someone not using --enable-local-rust.

[aidanhs]

Ok I have a more solid understanding of the issue. Currently, a stage0-out libstd will never link against jemalloc. stage1-out is left to its own devices, linking against the default allocator for the compiler that's just been created (stage0-out) which is jemalloc if enabled, system otherwise.

However, a local rebuild uses --cfg stage1 for building the stage0 libstd, which means it uses the default allocator for the local rust. This tends to be jemalloc...except we've disabled jemalloc, so it hasn't been built!

Something along the lines of #37975 seems like the right approach, but I think some elaboration around the logic in this area is necessary in comments or something.