Cannot build for mips and armhf with jemalloc

[herrernst]

I tried cross-compiling to mips, getting jemalloc errors when linking:

rustc hello.rs --target=mips-unknown-linux-gnu -C linker=mips-openwrt-linux-gcc
'generic' is not a recognized processor for this target (ignoring processor)
...
'generic' is not a recognized processor for this target (ignoring processor)
error: linking with `mips-openwrt-linux-gcc` failed: exit code: 1
note: "mips-openwrt-linux-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:843: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_insert':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_remove':

... and it goes on.
When adding this to main.rs it works

#![feature(alloc_system)]
extern crate alloc_system;

Error happens with stable 1.5, beta, latest nighty. I'm using the linker from OpenWRT SDK which identifies as "mips-openwrt-linux-gcc (OpenWrt/Linaro GCC 4.8-2014.04 r46450) 4.8.3"

Also I get a similar error when trying to compile for armhf:

rustc hello.rs --target=arm-unknown-linux-gnueabihf -C linker=~/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc
error: linking with `/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc` failed: exit code: 1
note: "/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
collect2: error: ld returned 1 exit status

error: aborting due to previous error

Here, I am using gcc from Raspberry Pi tools.

[MagaTailor]

Looks like a simple case of missing symbols - probably means the rlibs you're trying to use were built and linked against incompatible libraries.

EDIT:
You mention rpi - that immediately incriminates old glibc.

[herrernst]

I've downloaded the rust libs from http://static.rust-lang.org/dist/ndex.html, e.g. http://static.rust-lang.org/dist/2016-01-15/rust-std-nightly-mips-unknown-linux-gnu.tar.gz

find . -name "*jemalloc*"
./lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib
./lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib
./lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib

Is this the wrong approach? Should I build them myself?

[MagaTailor]

Yes, cause you're using a slightly old distro. If you tried it on one of the Odroids that would work as expected.

[alexcrichton]

Interesting! This is likely because we're using a pretty new toolchain to compile the MIPS nightlies, and we may need to downgrade that. This may be the same case for the ARM nightlies as well.

[lilianmoraru]

@alexcrichton Or just provide more options?
Edit:
Actually, yes, downgrading might be a better option.

[MagaTailor]

@herrernst

If you try to provide your own rlibs just be careful to not bump into this issue, relevant comment: #30363 (comment)

[herrernst]

Thanks. I'm quite new to rust. Cross-compiling std doesn't seem to be straightforward, so I'm just using the system allocator for now.
Update: I've now compiled Rust with mips target, and then using jemalloc also works.

[lilianmoraru]

@herrernst So, can the guys close this issue?
Who is interested, here is more on this issue.

[herrernst]

I think it should be considered downgrading the build chain for the prebuilt std libs, as alexcrichton has said, because it would be nice for example to use the standard Raspberry Pi toolchain without having to need compiling Rust yourself.

[joerg-krause]

@herrernst This is my approach https://github.com/joerg-krause/rust-cross-libs to cross-compile std for an ARMv5 target (which is an custom target), because I did not wanted to cross-bootstrap the whole Rust toolchain.

EDIT: insert link

[alexcrichton]

I would indeed like to assuage this issue if at all possible, but I also want to try to drill down here and get some specifics about how to move forward. What I believe to be the problem is that we're using "too new" of a C compiler when compiling jemalloc, and I suspect that this may be fixable by just using an older C compiler. With that in mind, first here's what we're using on the bots for building our nightlies:

• Ubuntu 15.10
• The ARM compiler is the gcc-arm-linux-gnueabi package with the version 4:5.2.1-1
• The MIPS compiler is the gcc-5-mips-linux-gnu package with the version 5.2.1-22ubuntu2mips1~ubuntu15.10.1cross1

Both of those gcc versions are indeed very new, but I wouldn't be 100% sure of how to downgrade, either, unfortunately. I suspect what actually needs to be downgraded is the glibc version for each platform (it just tends to be in step with a gcc version). With this all in mind, I think there's a few courses of action here:

1. Figure out if there's a "standard toolchain" here that we can use. Can we pick some older version of gcc/glibc which works for almost everyone, get some compilers for that, and use those to build nightlies?
2. Disable jemalloc by default on these platforms for now. This is, I believe, the only C code left in the standard library by default, so we should in theory erase that level of glibc portability issues if we stop including any C code.
3. Document what toolchains we're compiling jemalloc with and say that they're the minimum required version.

I'd personally be in favor of 1 or 2, but I unfortunately don't know what to do with option 1...

---

As a side note as well, it looks like the OpenWRT SDK uses uclibc? Does that strive for binary ABI compatibility with glibc? If not then there may be worse consequences mixing with the mips-unknown-linux-gnu target if the standard library doesn't agree with the uclibc library about various constant values and such. To solve that problem we could provide a mips-unknown-linux-uclibc, however.

[lilianmoraru]

On Ubuntu 15.10 the oldest GCC that can be installed is GCC 4.7.
Usually, if there is a package for GCC [version], than there are also available cross-compilers with the same versions.
So, instead of installing gcc-arm-linux-gnueabi it should just be changed into gcc-4.7-arm-linux-gnueabi and instead of gcc-arm-linux-gnueabihf -> gcc-4.7-arm-linux-gnueabihf.
GCC will pick the correct glibc on its own, nothing else to do here.
Note that the binary is actually called arm-linux-gnueabi-gcc-[version] and care has to be taken that the link arm-linux-gnueabi-gcc points to the correct version(It could point to the wrong version if another version was installed before this), in this case 4.7.

To the second question:
It should not be ok to use gcc that uses glibc to compile for a target that actually has all its libraries compiled against uclibc. The API might look the same and the software might compile but it is enough to have 1 more variable or the variables arrangement to be different and there will be some crazy bugs because of the memory layout.

Here is a comparison - you can see that uclibc doesn't even have a stable ABI.

ref Document what toolchains we're compiling jemalloc with and say that they're the minimum required version. - The archive name could probably include the compiler that was used, the same way it indicates the target triple.

[alexcrichton]

Thanks for the answer @lilianmoraru! The pointer for the gcc-4.7-arm-linux-gnueabihf package looks like it works for me, so all we need to do is test out it still compiles a toolchain ok and then I'd be fine updating the image.

For MIPS we have a custom PPA from @angelsl, so unfortunately there isn't a standard package for 4.7 (I believe) just yet. We can likely try to find one, however, or make it ourselves.

Also thanks for explaining uclibc, that matches what I'd suspect. I'm a little surprised that it doesn't even have a stable ABI (could make, e.g., porting liblibc to it a little difficult), but alas! In that case anyway @herrernst I don't think you can use our MIPS nightlies with the OpenWRT toolchain because there's a mismatch with the libc versions.

[herrernst]

Thanks for all the information and feedback work. One thing I'm not yet quite sure of: If a build a local std crate with the compiler from the OpenWRT toolchain, should it work then? It then links against uclibc, doesn't it? I picked some random Rust project, hyper, and the example server (albeit without OpenSSL for now) seems to run okay.