Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamically-linked and/or native musl targets #40113

Merged
merged 14 commits into from
Aug 23, 2017

Conversation

smaeul
Copy link
Contributor

@smaeul smaeul commented Feb 26, 2017

These changes allow native compilation on musl-based distributions and the use of dynamic libraries on linux-musl targets. This is intended to remove limitations based on past assumptions about musl targets, while maintaining existing behavior by default.

A minor related bugfix is included.

@rust-highfive
Copy link
Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @eddyb (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

@eddyb
Copy link
Member

eddyb commented Feb 26, 2017

r? @alexcrichton

@rust-highfive rust-highfive assigned alexcrichton and unassigned eddyb Feb 26, 2017
Copy link
Member

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me! Just some minor comments but otherwise I think this is good to go.

fn crt_static(&self, target: &str) -> Option<bool> {
self.config.target_config.get(target)
.and_then(|t| t.crt_static)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an idea actually, I think we can leverage this for MSVC as well. The bin/rustc.rs script above has a hard-coded branch for MSVC but I think we could change that here. Could this code be updated with something like:

.or_else(|| {
    if target.contains("msvc") {
        Some(true)
    } else {
        None
    }
})

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since +crt-static was previously unconditional for msvc, I left it that way when moving the check over.

if build.musl_root(target).is_none() && build.config.build == *target {
let target = build.config.target_config.entry(target.clone())
.or_insert(Default::default());
target.musl_root = Some("/usr".into());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment her to what this block is doing?

@smaeul
Copy link
Contributor Author

smaeul commented Mar 5, 2017

While working on getting these two details sorted out, I tried to use the stage2 rustc to bootstrap from again. However, it was getting immediately SIGKILL'd even with --help or --version. readelf told me it had no .text section.

Because src/librustc_back/target/linux_musl_base.rs adds -nostdlib to the linker arguments (to support building on glibc-targeting toolchains), the crt*.o startup files are not getting added, ld does not find the _start symbol, and -Wl,-O1 causes it to optimize out the entire binary.

We need to conditionally pass -nostdlib to the linker, just like we conditionally provide the startup files. However, pre_link_args is used by several other targets, so I'm not sure what to change.

I will rebase and take care of the other issues, though.

@smaeul smaeul force-pushed the native-musl branch 3 times, most recently from b052295 to 706fc55 Compare March 6, 2017 00:20
Copy link
Member

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch yeah that sounds like a bad issue. That's not fixed up in this PR, right? If so, did you want to include it here or postpone for a future PR? (I'm fine with either)

@@ -106,14 +106,17 @@ pub fn std_link(build: &Build,
t!(fs::create_dir_all(&libdir));
add_to_sysroot(&out_dir, &libdir);

if target.contains("musl") && !target.contains("mips") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I thought we were replacing the contains("mips") annotations with a different one? Otherwise this may end up breaking the mips-musl build :(

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was my understanding that we wanted to support both +crt-static and -crt-static for user code with any build of rustc. The crt_static option in config.toml only controls if rustc itself is statically or dynamically linked.

If we want to support static linking (especially with a glibc-based toolchain) on a target, we need musl-root and the startup files for that target.

@@ -157,8 +157,15 @@ pub fn check(build: &mut Build) {
panic!("the iOS target is only supported on OSX");
}

// Make sure musl-root is valid if specified
if target.contains("musl") && !target.contains("mips") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like above, the mips check being removed here may break that build

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, mips would be dynamic-only without this change

@alexcrichton
Copy link
Member

@bors: r+

Hm ok, makes sense. Let's see if mips builds break

@bors
Copy link
Contributor

bors commented Mar 6, 2017

📌 Commit 706fc55 has been approved by alexcrichton

@smaeul
Copy link
Contributor Author

smaeul commented Mar 6, 2017

In regards to -nostdlib completely breaking stage2, there are three different situations I see we're trying to support here:

  1. Dynamic linking with a musl toolchain
  2. Static linking with a musl toolchain
  3. Static linking with a glibc toolchain

Cases 1 and 2 can use the toolchain-provided crt*.o, libc, and libgcc_s/libunwind, so they do not need any files copied over, and they do not need any extra linker arguments. Case 3 requires providing our own crt*.o, libc.a, and libunwind.a, and it requires -nostdlib to suppress the default toolchain versions.

We can make case 2 look like case 3, since static linking is self-contained, but it is a bit of a hack (e.g. using /usr for musl-root will lock libc version to the one we copied into the sysroot when building rustc). Making case 1 look like case 3 is even more difficult and fragile. Sharing the same TargetOptions among all three is just not going to work.

I see three ways to resolve this. I don't know which is most preferable.

  1. More completely distinguish between cases 1 and {2,3}. This PR makes manually providing the startup files to the linker conditional on +crt-static. It might work to make -nostdlib also conditional on +crt-static.
  2. Distinguish between cases {1,2} and 3. If we have a way to know if we are linking with a foreign toolchain, we can drop all of the linker flags/files for case 2. Then musl-root would only be used for case 3.
  3. Drop built-in support for case 3, and require the user to have an appropriate linker (musl-gcc or a cross toolchain) to cross-link. Then we can remove the special-casing and musl-root entirely.

If I completely remove -nostdlib and the target files (unconditionally), I can build a complete stage2 with crt_static=true in config.toml. Building a stage1 with crt_static=false in config.toml also works, but fails in stage2-std trying to link shared libraries against a static libunwind. I have not investigated this yet, but it seems like a separate issue.

frewsxcv added a commit to frewsxcv/rust that referenced this pull request Mar 8, 2017
Support dynamically-linked and/or native musl targets

These changes allow native compilation on musl-based distributions and the use of dynamic libraries on linux-musl targets. This is intended to remove limitations based on past assumptions about musl targets, while maintaining existing behavior by default.

A minor related bugfix is included.
@frewsxcv
Copy link
Member

frewsxcv commented Mar 8, 2017

Looks like there's a failure here:

https://travis-ci.org/rust-lang/rust/jobs/209064374

@bors r-

@alexcrichton
Copy link
Member

@smaeul oh AFAIK we don't support static linking with glibc at all today (or at least if it looks like we do it wasn't intended)

@codyps
Copy link
Contributor

codyps commented Mar 8, 2017

@alexcrichton IIUC, "Static linking with a glibc toolchain" means, in this context "using a glibc toolchain to statically link to musl" (which, again IIUC, is the only currently supported mode prior to this PR)

@smaeul
Copy link
Contributor Author

smaeul commented Mar 8, 2017

@alexcrichton I could have been more clear there. I meant (for case 3 above) statically linking to musl with a toolchain that targets glibc, i.e. where musl-gcc or ${ARCH}-linux-musl-gcc is not available. As far as I can tell, that's the only case where musl-root or -nostdlib is necessary.

@alexcrichton
Copy link
Member

Oh right yeah that makes sense.

In reality we don't need a glibc toolchain at all, the intention is that we can produce a musl binary at any time. All we need is a linker really. But yeah I think the best solution is not passing -nostdlib if we're compiling with -C target-feature=-crt-static

@smaeul
Copy link
Contributor Author

smaeul commented Mar 13, 2017

The build failures here look to be because the tests for mips are not expecting a musl-root, so the default (/usr/local) is being used. The configure script and test infrastructure will need to be updated to add an option for this and generate the appropriate libraries.

@alexcrichton
Copy link
Member

@smaeul the configuration is in tree actually, mind updating that to pass the right configure args?

@alexcrichton
Copy link
Member

ping @smaeul, any updates here?

@smaeul
Copy link
Contributor Author

smaeul commented Mar 24, 2017

About figuring out when to pass -nostdlib and crt*, I found the easiest way to get things working both shared and static was to remove them entirely. In other words, if we just let the linker find libc and the startup files, we don't need any overriding at all (no linker flags, no musl root), as long as the linker knows where to find the files. And if the compiler is native, an appropriate cross linker exists, or musl-gcc exists, that is a valid assumption. Unfortunately, it breaks current behavior -- I don't know what's acceptable here.

Otherwise, we'll need to add some sort of conditional expression to src/librustc_back/target/linux_musl_base.rs, and I don't know how to do that.

I haven't worked on the CI/configure issue because I'm not sure how good of an idea it is to get this "working" but at the same time completely broken. I'd like to try to get the linking issue fixed first.

@alexcrichton
Copy link
Member

Yeah the purpose of the musl target today is that you don't need to have musl-gcc or a similar wrapper installed, so we'll need to preserve the capability somehow.

@jirutka
Copy link

jirutka commented Apr 7, 2017

I’ve tried to build rustc in Alpine Linux with this patch and I hit few issues.

At first, there’s a relevant part of the build script:

pkgver=1.16.0
_stage0dir="$srcdir/stage0"

# Don't set wrong LD_LIBRARY_PATH, we will rather set it manually when
# invoking make.
sed -i /LD_LIBRARY_PATH/d src/bootstrap/bootstrap.py

# Precompiled rustc and rust-std from VoidLinux.
cp -flr "$srcdir"/rustc-$pkgver-*/rustc/* \
        "$srcdir"/rust-std-$pkgver-*/rust-std-*/* \
        "$srcdir"/cargo-*-*/cargo/* \
        "$_stage0dir"/  

# Note: We use llvm 3.8.1 for now.
./configure \
        --build="$_ctarget" \
        --host="$_ctarget" \
        --prefix="/usr" \
        --release-channel="stable" \
        --enable-local-rust \
        --local-rust-root="$_stage0dir" \
        --llvm-root="/usr" \
        --musl-root="/usr" \
        --enable-rustbuild \
        --enable-vendor \
        --disable-docs \
        --disable-jemalloc \
        --disable-rpath

# Set LD_LIBRARY_PATH, so rustc in stage0 can find correct libs.
LD_LIBRARY_PATH="$_stage0dir/lib" \
        make RUST_BACKTRACE=1 RUST_CRT_STATIC="false" VERBOSE=1

LD_LIBRARY_PATH="$_stage0dir/lib" \
        make install DESTDIR="$pkgdir"

I’ve removed -nostdlib from src/librustc_back/target/linux_musl_base.rs:

-    // Make sure that the linker/gcc really don't pull in anything, including
-    // default objects, libs, etc.
-    base.pre_link_args.push("-nostdlib".to_string());

The build succeeds, but rustc fails when trying to compile anything:

$ rustc hello_world.rs
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-Wl,-(" "-m64" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "hello_world.0.o" "-o" "hello_world" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/librand-709b996bacb8eb07.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcollections-df1635bc25c3e592.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_unicode-00ee5191e548d906.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-1e0a4dc78495337e.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-a1562f522536bd3b.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_system-bde4c172625df17f.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-bf68330d3f5e10dd.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-963774836b754896.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-9802cab648d12028.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
  = note: /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o: In function `_start':
          crt1.c:(.text+0x0): multiple definition of `_start'
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/Scrt1.o:Scrt1.c:(.text+0x0): first defined here
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o: In function `_start_c':
          crt1.c:(.text._start_c+0x0): multiple definition of `_start_c'
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/Scrt1.o:Scrt1.c:(.text._start_c+0x0): first defined here
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o: In function `_init':
          /home/buildozer/aports/main/musl/src/musl-1.1.16/crt/x86_64/crti.s:4: multiple definition of `_init'
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/crti.o:/home/buildozer/aports/main/musl/src/musl-1.1.16/crt/x86_64/crti.s:4: first defined here
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o: In function `_fini':
          /home/buildozer/aports/main/musl/src/musl-1.1.16/crt/x86_64/crti.s:4: multiple definition of `_fini'
          /usr/lib/gcc/x86_64-alpine-linux-musl/6.3.0/../../../../lib/crti.o:/home/buildozer/aports/main/musl/src/musl-1.1.16/crt/x86_64/crti.s:4: first defined here
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `collections::vec::{{impl}}::into_boxed_slice<u8>':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/raw_vec.rs:517: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `alloc::heap::deallocate':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `std::sys::imp::mutex::{{impl}}::unlock':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/mutex.rs:72: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `core::result::unwrap_failed<std::ffi::c_str::NulError>':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `drop':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o):/home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: more undefined references to `_Unwind_Resume' follow
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `std::sys::imp::backtrace::tracing::imp::write':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42: undefined reference to `_Unwind_Backtrace'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `std::sys::imp::backtrace::tracing::imp::write::trace_fn':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:62: undefined reference to `_Unwind_GetIPInfo'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:83: undefined reference to `_Unwind_FindEnclosingFunction'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:93: undefined reference to `_Unwind_Resume'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:103: undefined reference to `_Unwind_Resume'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:103: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `alloc::heap::deallocate':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o): In function `std::sys::imp::os_str::{{impl}}::fmt':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-d73d432e81fd8b19.rlib(std-d73d432e81fd8b19.0.o):/home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/arc.rs:768: more undefined references to `_Unwind_Resume' follow
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::imp::find_eh_action':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:264: undefined reference to `_Unwind_GetLanguageSpecificData'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:266: undefined reference to `_Unwind_GetIPInfo'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::dwarf::eh::find_eh_action':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/dwarf/eh.rs:69: undefined reference to `_Unwind_GetRegionStart'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::imp::rust_eh_personality':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:172: undefined reference to `_Unwind_SetGR'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:173: undefined reference to `_Unwind_SetGR'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:174: undefined reference to `_Unwind_SetIP'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::imp::find_eh_action::{{closure}}':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:272: undefined reference to `_Unwind_GetTextRelBase'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:273: undefined reference to `_Unwind_GetDataRelBase'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::imp::cleanup':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:100: undefined reference to `_Unwind_DeleteException'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `panic_unwind::imp::panic':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:83: undefined reference to `_Unwind_RaiseException'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `core::ops::FnOnce::call_once':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:272: undefined reference to `_Unwind_GetTextRelBase'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-85f96f5a1caa812c.rlib(panic_unwind-85f96f5a1caa812c.0.o): In function `core::ops::FnOnce::call_once':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:273: undefined reference to `_Unwind_GetDataRelBase'
          collect2: error: ld returned 1 exit status

We can see three problems here:

  1. Wrong absolute paths coming from build environment into the stack trace. This is an unrelated error, I’ll create a separate issue for it.
  2. crt1.c:(.text+0x0): multiple definition of '_start'
  3. undefined reference to '_Unwind_Resume'

I’ve fixed the second problem by removing the following snippet from src/librustc_back/target/linux_musl_base.rs:

-    // When generating a statically linked executable there's generally some
-    // small setup needed which is listed in these files. These are provided by
-    // a musl toolchain and are linked by default by the `musl-gcc` script. Note
-    // that `gcc` also does this by default, it just uses some different files.
-    //
-    // Each target directory for musl has these object files included in it so
-    // they'll be included from there.
-    base.pre_link_objects_exe.push("crt1.o".to_string());
-    base.pre_link_objects_exe.push("crti.o".to_string());
-    base.post_link_objects.push("crtn.o".to_string());

And the third problem by removing this: (This actually hasn’t fixed it, removing base.crt_static_default = true; and so linking dynamically by default did)

-    base.pre_link_args.push("-Wl,-(".to_string());
-    base.post_link_args.push("-Wl,-)".to_string());

After these changes we end up just with base.pre_link_args.push("-Wl,--eh-frame-hdr".to_string()); (which is maybe not even needed), otherwise it’s the same as linux_base. And rustc finally works!

$ rustc -C target-feature=-crt-static hello_world.rs
$ ./hello_world
Hello, world!
$ ldd hello_world
        /lib/ld-musl-x86_64.so.1 (0x2ecbf67a000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x2ecbf240000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x2ecbf67a000)

I’m not sure if it’s really intentional that it links dynamically by default now, but I don’t mind this. (Pardon, I also removed base.crt_static_default = true; and forgot about it.)

However, statical linking is now broken:

$ rustc -C target-feature=+crt-static hello_world.rs
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "hello_world.0.o" "-o" "hello_world" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/librand-36efc09e6c3aa7b6.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcollections-64ae13b816203712.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_unicode-a6dea33a6a8b7481.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-52493d6bc7a4edfb.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-8ffb79da497eb47f.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc_system-35e8b7ebb64c18bd.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-2b2567c2ae2ad7d0.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-6588f2833738184c.rlib" "/usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-5ef8fa488c4b9f32.rlib"
  = note: /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `collections::vec::{{impl}}::into_boxed_slice<u8>':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/raw_vec.rs:517: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `alloc::heap::deallocate':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `core::result::unwrap_failed<std::ffi::c_str::NulError>':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `std::sys::imp::mutex::{{impl}}::unlock':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/mutex.rs:72: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `drop':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o):/home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: more undefined references to `_Unwind_Resume' follow
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `std::sys::imp::backtrace::tracing::imp::write':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:42: undefined reference to `_Unwind_Backtrace'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `std::sys::imp::backtrace::tracing::imp::write::trace_fn':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:62: undefined reference to `_Unwind_GetIPInfo'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:83: undefined reference to `_Unwind_FindEnclosingFunction'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:93: undefined reference to `_Unwind_Resume'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:103: undefined reference to `_Unwind_Resume'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:103: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `alloc::heap::deallocate':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o): In function `std::sys::imp::os_str::{{impl}}::fmt':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/heap.rs:113: undefined reference to `_Unwind_Resume'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-86059d53218552f5.rlib(std-86059d53218552f5.0.o):/home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/liballoc/arc.rs:768: more undefined references to `_Unwind_Resume' follow
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::imp::find_eh_action':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:264: undefined reference to `_Unwind_GetLanguageSpecificData'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:266: undefined reference to `_Unwind_GetIPInfo'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::dwarf::eh::find_eh_action':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/dwarf/eh.rs:69: undefined reference to `_Unwind_GetRegionStart'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::imp::rust_eh_personality':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:172: undefined reference to `_Unwind_SetGR'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:173: undefined reference to `_Unwind_SetGR'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:174: undefined reference to `_Unwind_SetIP'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::imp::find_eh_action::{{closure}}':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:272: undefined reference to `_Unwind_GetTextRelBase'
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:273: undefined reference to `_Unwind_GetDataRelBase'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::imp::cleanup':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:100: undefined reference to `_Unwind_DeleteException'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `panic_unwind::imp::panic':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:83: undefined reference to `_Unwind_RaiseException'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `core::ops::FnOnce::call_once':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:272: undefined reference to `_Unwind_GetTextRelBase'
          /usr/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-12770ed0830004c4.rlib(panic_unwind-12770ed0830004c4.0.o): In function `core::ops::FnOnce::call_once':
          /home/jirutjak/aports/testing/rust/src/rustc-1.16.0-src/src/libpanic_unwind/gcc.rs:273: undefined reference to `_Unwind_GetDataRelBase'
          collect2: error: ld returned 1 exit status

error: aborting due to previous error

(Yes, I’ve removed that pesky check that rejects +crt-static in stable…)

So we probably need to make some options in linux_musl_base.rs conditional based on crt-static…?


@shizmob tried different approach, he added:

+    // To prevent some nasty linking errors, link in libgcc_s here.
+    base.pre_link_args.push("-lgcc_s".to_string());

This also works, but haven’t tested compiling with +crt-static yet.

/cc @japaric

@japaric
Copy link
Member

japaric commented Apr 7, 2017

So we probably need to make some options in linux_musl_base.rs conditional based on crt-static…?

That was my conclusion from #38075 but I'm not too fond of the idea. #40018 already makes the link_args fields conditional wrt to -Z linker-flavor. If we also make it conditional wrt crt-static then we are going to double the number of possible combinations. I can't think of a better option atm.

bors added a commit that referenced this pull request Aug 25, 2017
Do not assume libunwind.a is available on musl

Fixes #40113, #44069, and clux/muslrust#16.

libunwind.a is not copied from musl_root, so it must be integrated into the unwind crate.
@corbinu
Copy link

corbinu commented Aug 29, 2017

I don't know if this is the right place to ask but how do I build this on Alpine? I tried cloning master and following the readme and then did this ./x.py --build=x86_64-unknown-linux-musl but it errors out when downloading the assets

@aidanhs aidanhs mentioned this pull request Oct 3, 2017
bors added a commit that referenced this pull request May 11, 2018
Use the correct crt*.o files when linking musl targets.

This is supposed to support optionally using the system copy of musl
libc instead of the included one if supported. This currently only
affects the start files, which is enough to allow building rustc on musl
targets.

Most of the changes are analogous to crt-static.

Excluding the start files is something musl based distributions usually patch into their copy of rustc:
  - https://github.com/alpinelinux/aports/blob/eb064c8/community/rust/musl-fix-linux_musl_base.patch
  - https://github.com/voidlinux/void-packages/blob/77400fc/srcpkgs/rust/patches/link-musl-dynamically.patch

For third-party distributions that not yet carry those patches it would be nice if it was supported without the need to patch upstream sources.

## Reasons
### What breaks?
Some start files were missed when originally writing the logic to swap in musl start files (gcc comes with its own start files, which are suppressed by -nostdlib, but not manually included later on). This caused #36710, which also affects rustc with the internal llvm copy or any other system libraries that need crtbegin/crtend.

### How is it fixed?
The system linker already has all the logic to decide which start files to include, so we can just defer to it (except of course if it doesn't target musl).

### Why is it optional?
In #40113 it was first tried to remove the start files, which broke compiling musl-targeting static binaries with a glibc-targeting compiler. This is why it eventually landed without removing the start files. Being an option side-steps the issue.

### Why are the start files still installed?
This has the nice side-effect, that the produced rust-std-* binaries can still be used by on a glibc-targeting system with a rustc built against glibc.

## Does it work?
With the following build script (using [musl-cross-make](https://github.com/richfelker/musl-cross-make)): https://shadowice.org/~mixi/rust-musl/build.sh, I was able to cross-compile a musl-host musl-targeting rustc on a glibc-based system. The resulting binaries are at https://shadowice.org/~mixi/rust-musl/binaries/. This also requires #50103 and #50104 (which are also applied to the branch the build script uses).
@lxylbds
Copy link

lxylbds commented Feb 21, 2020

Libc dynamic library problem:
rustc -C prefer-dynamic -C linker="aarch64-linux-gnu-gccc" --target=aarch64-unknown-linux-musl -C link-args="--sysroot=/tmp/sysroots/aarch64-linux-gun-elf/ -Wl,-Bdynamic -lc -lulibs -lsrv_fs -lsrv_net -lgcc -Wl,--dynamic-linker=/lib/ld.so.elf" -C target-feature=crt-static hello.rs

It builds sucessfully,but runs failed.

It looks like libc.so is dynamically linked, but the function"calloc"(which is a member of libc) is statically linked. This caused libc to fail to call calloc during loading. How to solve this problem?

ps:There are both libc.a and libc.so in the compilation environment. Could rust links libc.so only, without libc.a linked ?

@Others
Copy link
Contributor

Others commented Feb 21, 2020

@lxylbds this is a pretty ancient PR. Maybe try asking on the Rust discord? Or perhaps create a new issue if you think this is a bug?

mguegan added a commit to DDd-Devops/cryptgeon that referenced this pull request May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-review Status: Awaiting review from the assignee but also interested parties.
Projects
None yet
Development

Successfully merging this pull request may close these issues.