Port `compiler-rt` intrinsics to Rust

[japaric]

Rationale

Porting compiler-rt to Rust is one of the remaining obstacles towards the intersection of our "on the fly compilation of std" and "rust everywhere" dreams.

For our goal of "on the fly compilation of std" (or any other set of "standard" crates), we want to minimize the number of C dependencies required to build std as these complicate the compilation process: users need a C (cross) compiler and we (rust-lang/rust) have to support (cross) compiling each of these C dependencies, which are wrapped in crates, to every target (even custom ones!) that downstream users may use -- this last part leads to complicated build.rs scripts that need to handle conditional compilation logic and deal with obscure gcc flags . On Linux, there are three C dependencies that we have to deal with: backtrace, jemalloc and compiler-rt. backtrace and jemalloc are optional on Linux and not available on some other platforms but compiler-rt is mandatory on most (all?) the targets we support.

This issue is about porting compiler-rt to Rust. Once ported, we can say goodbye to its complicated build.rs and make std easier to (cross) compile! An extra advantage is that, with this change, the compiler-rt intrinsics will receive the same optimizations as the std crate. This is not done today because it would make the build.rs even more complicated: flags like -march and -mcpu would have to be conditionally passed to gcc according to $TARGET.

The process

The goal is simple: We have to port each and every compiler-rt intrinsic along with their tests to the rustc-builtins crate.

The process could go two ways:

1. Using a "wholesale" approach: We can develop the new rustc-builtins crate out of tree: porting intrinsics and unit tests over time. Once all the intrinsics and unit tests are ported we can replace the in-tree rustc-builtins crate with the out-of-tree one and have the buildbots (and probably crater) serve as an integration test.
2. Using an incremental approach: We can rely on the fact that gcc_s provides the same intrinsics as compiler-rt and simply remove rustc-builtins's build.rs. This effectively means that rustc-builtins will no longer provide any intrinsic and that std programs will instead use gcc_s' intrinsics. Then we can start porting intrinsics + unit tests and adding them to rustc-builtins one by one. Each time a intrinsic is added, Rust programs will start using that intrinsic instead of the gcc_s' one.

The advantage of (2) is that we get an std that's easy to cross compile early on (because rustc-builtins is essentially empty!). Its disadvantage is that no_std programs which don't link to gcc_s and that depend on compiler-rt intrinsics will abruptly stop to compile because of linker errors (undefined reference to $intrinsic).

Prioritization

Some Rust targets use more or different intrinsics than others. If we take the incremental approach mentioned in the previous section, it makes sense to prioritize porting the intrinsics required by tier-1 platforms. This gist contains a "hack" to yields the list of intrinsics required to link rustc for a certain target plus lists of intrinsics generated with this "hack" for a few (right now, two) targets. The lists contained therein can be used to decide which intrinsics to prioritize.

Drawbacks

On each LLVM upgrade, we would have to carefully check if any compiler-rt intrinsics have been added (we already have to do this today) and the upgrade would be blocked on porting those new intrinsics to Rust first (this is the extra work that this change would create).

Unresolved questions

• Pick an approach to implement this change.

compilerrt_abort

Some intrinsics can fail (e.g. absvdi2). compiler-rt handles the failures by "aborting" -- it calls a compilerrt_abort() function. Should this function be ported to Rust? Or should the intrinsics, instead, panic! on failure?

---

cc @alexcrichton @brson

[TimNN]

I'm probably missing some information about the details of compiler-rt, but wouldn't a third way to go be to implement some intrinsic in rust and then remove the corresponding c file? Wouldn't this give us (some of) the advantage of (2) without the disadvantage?

[japaric]

@TimNN We can do that too but it doesn't give us the advantage of (2) because we would still rely on a external C (cross) compiler to compile all the intrinsics that haven't yet been ported to Rust. The advantage of (2) is that we can immediately get rid of the dependency on a external C (cross) compiler.

[TimNN]

Ah, I see, I should probably have read that section more carefully; thanks for clearing that up.

[japaric]

This effectively means (..) that std programs will instead use gcc_s' intrinsics.

Note that (at least on Linux) this already happens today, will still happen until #35021 lands and probably has been the case for a long time. Evidence below:

#![feature(start)]

use std::ptr;

#[start]
fn start(_: isize, _: *const *const u8) -> isize {
    let x = unsafe {
        // to prevent the compiler from optimizing away the whole floating point operation
        ptr::read_volatile(0x0 as *const f64)
    };
    let y = 1.;
    let z = x + y;

    z as isize
}

$ rustc --target arm-unknown-linux-gnueabi -C linker=arm-linux-gnueabi-gcc -Z print-link-args foo.rs
"arm-linux-gnueabi-gcc" (..) "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-l" "compiler-rt"

Note: -l gcc_s appears before -l compiler-rt

$ arm-linux-gnueabi-objdump -Cd foo
(..)
00000834 <foo::start::h949dda06cb9c83c0>:
(..)
 86c:   ebffff6b        bl      620 <__aeabi_dadd@plt>
 870:   ebffff5e        bl      5f0 <__aeabi_d2iz@plt>
(..)

Note: @plt dynamic loading)

$ arm-linux-gnueabi-nm foo
(..)
         U __aeabi_d2iz@@GCC_3.5
         U __aeabi_dadd@@GCC_3.5
         U __aeabi_unwind_cpp_pr0@@GCC_3.5
(..)

Note: U (undefined) and @GCC_3.5 (symbol versioning)

The intrinsics for floating point operations come from libgcc_s.so rather than from libcompiler-rt.a.

[japaric]

Using a "wholesale" approach: We can develop the new rustc-builtins crate out of tree

I've started with this approach in this repository. I've only ported a few intrinsics, mainly the ones that are needed for ARM Cortex-M targets to get basic stuff like ptr::copy_nonoverlapping working, but I'm already using it in one of my projects.

[Amanieu]

I think we should also include some other functions that are required by Rust but not provided by compiler-rt:

• fmod and fmodf
• memcmp, memcpy, memmove and memset

These should be defined as weak symbols so that an optimized implementation from libc can be used if available.

[japaric]

@Amanieu Sounds like a good idea to me. But, is there a mechanism to mark a symbol as weak from within Rust? I've done it before for executables but using a linker script; that won't work for rlibs because building those doesn't involve the linker.

[retep998]

On Windows, the PE/COFF specification describes weak externals which can be used for the purpose of defining symbols such as memcpy and whatnot so it can work without the CRT, but if you do link the CRT it'll use those instead.

“Weak externals” are a mechanism for object files that allows flexibility at link time. A module can contain an unresolved external symbol (sym1), but it can also include an auxiliary record that indicates that if sym1 is not present at link time, another external symbol (sym2) is used to resolve references instead.

If a definition of sym1 is linked, then an external reference to the symbol is resolved normally. If a definition of sym1 is not linked, then all references to the weak external for sym1 refer to sym2 instead. The external symbol, sym2, must always be linked; typically, it is defined in the module that contains the weak reference to sym1.

Weak externals are represented by a symbol table record with EXTERNAL storage class, UNDEF section number, and a value of zero. The weak-external symbol record is followed by an auxiliary record with the following format:

[japaric]

@retep998 Ok, that sounds like there might exists a LLVM thing to mark a symbol as "weak". I guess my actual question was: is there any Rust attribute to mark a function as weak?

[alexcrichton]

@japaric I'd personally prefer the incremental approach of just moving intrinsics from C to Rust over time. That way we can ensure that everyone keeps building locally, and we could even add cargo features perhaps to disable all C intrinsics so if all the Rust intrinsics suffice for your target you can get by with pure Rust anyway.

[shepmaster]

is there any Rust attribute to mark a function as weak?

Perhaps #[linkage = "extern_weak"]? For AVR work, I just used some custom assembly...

[japaric]

@shepmaster

#[linkage = "weak"] does the trick (on Linux at least):

#[linkage = "weak"]
#[no_mangle]
pub extern fn memfoo() {}

$ nm target/debug/libfoo.rlib
0000000000000000 W memfoo
0000000000000000 N __rustc_debug_gdb_scripts_section__

@alexcrichton Sounds good to me. One thing I like about having rustc-builtins in its own repo is that I've set up testing infrastructure (Travis + AppVeyor) to test the intrinsics for several targets (using QEMU), which I think is not trivial to integrate with the buildbots. Also, fast test times. Perhaps we can keep this advantage if we turn rustc-builtins into a submodule.

[retep998]

C:\msys64\home\Peter\test> dumpbin /symbols libweak.rlib
Microsoft (R) COFF/PE Dumper Version 14.00.24210.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file libweak.rlib

File Type: LIBRARY

COFF SYMBOL TABLE
000 00000000 SECT1  notype       Static       | .text
    Section length    0, #relocs    0, #linenums    0, checksum        0
002 00000000 SECT2  notype       Static       | .data
    Section length    0, #relocs    0, #linenums    0, checksum        0
004 00000000 SECT3  notype       Static       | .bss
    Section length    0, #relocs    0, #linenums    0, checksum        0
006 00000000 SECT4  notype       Static       | .text
    Section length    1, #relocs    0, #linenums    0, checksum  26D930A, selection    1 (pick no duplicates)
008 00000000 SECT4  notype ()    External     | memfoo
009 00000000 SECT5  notype       Static       | .xdata
    Section length    8, #relocs    0, #linenums    0, checksum CCAA009E, selection    5 (pick associative Section 0x4)
00B 00000000 SECT6  notype       Static       | .pdata
    Section length    C, #relocs    3, #linenums    0, checksum CCAA009E, selection    5 (pick associative Section 0x4)

String Table Size = 0x0 bytes
libweak.rlib : fatal error LNK1106: invalid file or disk full: cannot seek to 0x31A1

Doesn't look like it works on Windows. If it worked there would be a WeakExternal in there.

[alexcrichton]

@japaric I'd be down for that! I agree that moving CI off buildbot is always best :)