Support older versions of the ARM architecture

[japaric]

As of 2017-05-23 Rust officially supports only the ARMv6 and ARMv7 architectures. For ARM Linux, binary releases of std are provided for these targets. For bare metal ARM, there are built-in targets for ARM Cortex-M microcontrollers.

These are two cases that I know of that are currently unsupported:

• ARMv5 processors that run Linux
• ARMv4 processors (microcontrollers?) that operate without an OS

ARMv5 Linux

std

These targets want to use std but from the POV of LLVM these targets don't support atomics as there are no atomic load / store instructions in the instruction set. This leaves these targets with two options:

• Declare max-atomic-width as being 0. This removes atomic types like AtomicUsize from libcore making it impossible to build libstd as std depends on those atomic types. The built-in target armv5te-unknown-linux-gnueabi does this.

• Declare max-atomic-width as being 32. With this change libstd can be compiled but trying to link a std program will result in linker errors of the form: "undefined reference to __sync_fetch_and_foo". As there are no atomic instructions LLVM lower atomic operations to these intrinsics. These intrinsics will have to be provide by the user as there's no implementation for then in the compiler-rt project. One possible implementation for these intrinsics is to use Linux's kernel user helpers as suggested here.

uclibc

Some of these targets want to use uclibc instead of glibc because that's what their systems' OS uses. The complication here is that the libc crate does not support uclibc. However the libc crate does compile when compiled for an uclibc target, but it will actually assume that the target is using glibc. This is rather bad because uclibc and glibc don't have the same ABI so using the libc crate for an uclibc target can result in a segfault / crash at runtime.

Important: All std programs make use of the libc crate so all std programs are affected by this issue.

Bare metal ARMv4

I don't have details for this target but since it's a bare metal target it will work in no_std mode. A custom target will be required as there's no built-in target in rustc. The specification of that custom target can probably derived from the specification of an existing target. thumbv6m-none-eabi seems like a good base target:

$ rustc -Z unstable-options --print target-spec-json --target thumbv6m-none-eabi
{
  "abi-blacklist": [
    "stdcall",
    "fastcall",
    "vectorcall",
    "win64",
    "sysv64"
  ],
  "arch": "arm",
  "data-layout": "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64",
  "env": "",
  "executables": true,
  "features": "+strict-align",
  "is-builtin": true,
  "linker": "arm-none-eabi-gcc",
  "linker-flavor": "gcc",
  "llvm-target": "thumbv6m-none-eabi",
  "max-atomic-width": 0,
  "os": "none",
  "panic-strategy": "abort",
  "relocation-model": "static",
  "target-endian": "little",
  "target-pointer-width": "32",
  "vendor": ""
}

[thejpster]

What about Cortex-R? You see a few Cortex-R4s around in WiFi processors (I think there's a Cypress/Broadcom one, for example).

[japaric]

@thejpster I have never dealt with one of those, but I suppose that as long as you create a proper target specification file then things should just work? LLVM seems to support the architecture.

[jamesmunns]

@japaric @thejpster I know @uvekilledkenny is doing some bare metal work on ARMv4 targets in their Rust on GBA work here: https://github.com/Uvekilledkenny/rba - specifically with the ARM7TDMI the GBA uses.

Might be useful to talk to them about it

[japaric]

@thejpster This looks like a reasonable target specification for the Cortex-R4:

{
    "abi-blacklist": [
        "stdcall",
        "fastcall",
        "vectorcall",
        "win64",
        "sysv64"
    ],
    "arch": "arm",
    "data-layout": "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64",
    "env": "",
    "executables": true,
    "linker": "arm-none-eabi-ld",
    "linker-flavor": "ld",
    "llvm-target": "armv7r-none-eabi",
    "max-atomic-width": 32,
    "os": "none",
    "panic-strategy": "abort",
    "relocation-model": "static",
    "target-endian": "little",
    "target-pointer-width": "32",
    "vendor": ""
}

Atomics work in the sense that atomic ops get lowered to dmb instructions like the thumb targets.

[jannic]

About max-atomic-width on ARMv5: If I understand it correctly, the PR at rust-lang/compiler-builtins#115 isn't even necessary. LLVM already knows how to call the kernel helpers.

I tried it by just setting max-atomic-width to 64. (Not sure if 32 would be more appropriate, but the kernel helpers are available in 64 bit variants, wo why not?)

The resulting code compiling some simple rust program looks fine to me. Some random function, only selected because it's short:

00008014 <_ZN33_$LT$alloc..arc..Arc$LT$T$GT$$GT$9drop_slow17h2c0cacae4125c3f3E>:
    8014:       e92d4830        push    {r4, r5, fp, lr}
    8018:       e1a05000        mov     r5, r0
    801c:       e5954000        ldr     r4, [r5]
    8020:       e2840008        add     r0, r4, #8
    8024:       eb000187        bl      8648 <_ZN4core3ptr13drop_in_place17h11f8f40bb4e41bc7E>
    8028:       e5950000        ldr     r0, [r5]
    802c:       e3a01001        mov     r1, #1
    8030:       e2800004        add     r0, r0, #4
    8034:       eb01210c        bl      5046c <__sync_fetch_and_sub_4>
    8038:       e3500001        cmp     r0, #1
    803c:       18bd8830        popne   {r4, r5, fp, pc}
    8040:       eb0124af        bl      51304 <__sync_synchronize>
    8044:       e1a00004        mov     r0, r4
    8048:       e3a01028        mov     r1, #40 ; 0x28
    804c:       e3a02008        mov     r2, #8
    8050:       e8bd4830        pop     {r4, r5, fp, lr}
    8054:       ea004e8f        b       1ba98 <__rust_deallocate>

where __sync_fetch_and_sub_4 is:

0005046c <__sync_fetch_and_sub_4>:
   5046c:       e92d41f0        push    {r4, r5, r6, r7, r8, lr}
   50470:       e1a05000        mov     r5, r0
   50474:       e1a07001        mov     r7, r1
   50478:       e59f6028        ldr     r6, [pc, #40]   ; 504a8 <__sync_fetch_and_sub_4+0x3c>
   5047c:       e5954000        ldr     r4, [r5]
   50480:       e1a02005        mov     r2, r5
   50484:       e0441007        sub     r1, r4, r7
   50488:       e1a00004        mov     r0, r4
   5048c:       e1a0e00f        mov     lr, pc
   50490:       e12fff16        bx      r6
   50494:       e3500000        cmp     r0, #0
   50498:       1afffff7        bne     5047c <__sync_fetch_and_sub_4+0x10>
   5049c:       e1a00004        mov     r0, r4
   504a0:       e8bd41f0        pop     {r4, r5, r6, r7, r8, lr}
   504a4:       e12fff1e        bx      lr
   504a8:       ffff0fc0        .word   0xffff0fc0

So it just jumps to 0xffff0fc0 for its atomic operation, which is the location of the kernel user helper kuser_cmpxchg. (But please note that I know basically nothing about ARM assembler...)

Unfortunately, I can't say that rust works on armv5, yet, because I still have some issues which are probably unrelated to atomic operations. I'm still investigating of it's just me doing something stupid or if there's a real issue, I'll report on that later.

(My test platform is Lego Mindstorms EV3, which, according to https://en.wikipedia.org/wiki/Lego_Mindstorms_EV3#Overview, contains an ARM926EJ-S CPU.)

In conclusion, I think that rust-lang/rust@365ea80 was unnecessary and could be reverted.

[jannic]

And, as a quick update, using rust beta I actually get a binary which works on the Mindstorm. Just the default 'Hello world' from cargo new --bin, for now, and some trivial cloning of Arc.
(With current master I got some panics I don't understand, yet.)

[japaric]

LLVM already knows how to call the kernel helpers.

Maybe this changed recently due to the recent LLVM upgrade to 4.0. I certainly recall hitting the "undefined reference" errors when linking a std program for ARMv5. I'm grepping both llvm and compiler-rt sources and I don't the magic 0xffff0fc0 or an implementation that looks like your code; there's an implementation but it uses dmb which is not available on ARMv5, AFAIK. Are you sure that __sync_fetch_and_sub_4 symbol is not coming from a C library like libgcc?

cc @Amanieu ^

(With current master I got some panics I don't understand, yet.)

You could check if the LLVM version is different between master and beta (see rustc -Vv). Sometimes codegen bugs appear when LLVM upgrades occur.

[Amanieu]

@japaric These functions are provided by libgcc.

[whitequark]

But LLVM calling __sync_fetch_and_sub_4 is a perfectly desirable behavior! Just, these functions should be provided by libcompiler_builtins conditional on building for a non-none target, or otherwise by some rt crate like cortex-m-rt.

In a way atomics being always enabled in libcore but predicated on builtins support is a situation that parallels softfloat being always enabled in libcore but predicated on builtins support. We should treat both the same way IMO.

[jannic]

@japaric Does it matter if __sync_fetch_and_sub_4 is coming from compiler-rt or libgcc? In the end, it gets linked into the binary and works as expected. From my point of view, it's just part of the environment provided by -linux-gnueabi.
(This is not a rhetorical question: It's perfectly possible that I'm just missing something!)

Of course, for some hypothetical armv5-none-eabi target, another solution would be needed to provide atomics, but such a target doesn't exist yet, and if it were created, it would have its own target definition which could set max_atomic_width to zero.

[Amanieu]

The implementation of __sync_fetch_and_sub_4 in compiler-rt is ARMv6+ only. Only the libgcc version supports ARMv4/5. But you are right, we could just link with libgcc and atomics should work.

[jannic]

When you write 'we could just link with libgcc', does that mean it doesn't happen, now? Then, where does the code quoted above come from? It clearly uses the kernel user helpers, by jumping to 0xffff0fc0, so it should work on ARMv5. And it looks like that jump is neither in llvm nor in compiler-rt, so were does it come from?

[jannic]

While I was unable to follow the LLVM source code, using strace on rustc I saw that /usr/lib/gcc-cross/arm-linux-gnueabi/6/libgcc.a, which contains the __sync_* symbols, gets read during compilation.
This at least solves the mystery where those symbols get defined.

[japaric]

Does it matter if __sync_fetch_and_sub_4 is coming from compiler-rt or libgcc?

It matters in the sense that we shouldn't break linking of std programs in the future. I recall std programs not linking for ARMv5 before so maybe something changed in the libc crate and we now link to libgcc.a but we didn't before. It would be bad if that changed again and broke linking of the ARMv5 target.

I would prefer if the __sync_fetch_and_sub_4 symbols were defined in the compiler-builtins crate that way it would be possible to write std programs that don't depend on C code that use atomics. But I'm obviously biased :-).

[jannic]

Found the cause for the panic I got with nightly on ARMv5: rust-lang/rust#42314