Dangerous allocation linker error when using custom allocators

[pacmancoder]

I have faced the issues with the "dangerous relocation" linker errors when compiling the rust library and linking it to the main ESP8266 project (IDF-style ESP8266 RTOS SDK v3.3).

I have tried to overcome this issue with solution proposed by @lexxvir in the comment, substituting linker by the stript; Also tried to add -C link-args="-mlongcalls -Wa,--auto-lit-pools" to the RUSTFLAGS, tried to add adding the same args to the link args for IDF CMake file, but still no luck.

This issue kicks in when I try to perform any dynamic allocation after with alloc crate (e.g. boxing or pushing some data to the String)

Error output:

../../target/xtensa-esp8266-none-elf/release/libfirmware.a(firmware-5b7e9638ff0ed6ee.2qffuh8m3mnmhmu5.rcgu.o): In function `__rust_alloc':
2qffuh8m3mnmhmu5:(.text.__rust_alloc+0xb): dangerous relocation: l32r: literal target out of range (try using text-section-literals): .literal
../../target/xtensa-esp8266-none-elf/release/libfirmware.a(firmware-5b7e9638ff0ed6ee.2qffuh8m3mnmhmu5.rcgu.o): In function `__rust_dealloc':
2qffuh8m3mnmhmu5:(.text.__rust_dealloc+0xd): dangerous relocation: l32r: literal target out of range (try using text-section-literals): (.literal+0x4)
../../target/xtensa-esp8266-none-elf/release/libfirmware.a(firmware-5b7e9638ff0ed6ee.2qffuh8m3mnmhmu5.rcgu.o): In function `__rust_realloc':
2qffuh8m3mnmhmu5:(.text.__rust_realloc+0xf): dangerous relocation: l32r: literal target out of range (try using text-section-literals): (.literal+0x8)
collect2: error: ld returned 1 exit status

Currently, I have found a rather ugly solution, but at least it works:

I have explored the alloc crate sources and found that the allocation mechanism in rust is based on a few non-mangled functions, so I implemented them in my crate as direct calls to the custom allocator:

use idf_alloc::IdfAllocator;

use core::alloc::{GlobalAlloc, Layout};

#[global_allocator]
static GLOBAL_ALLOCATOR: IdfAllocator = IdfAllocator;

#[alloc_error_handler]
fn rust_idf_oom_error_handler(info: Layout) -> ! { panic!("Out of memory error!"); }

// The tricky part starts here

#[no_mangle]
pub unsafe fn __rust_alloc(size: usize, align: usize) -> *mut u8 {
    GLOBAL_ALLOCATOR.alloc( Layout::from_size_align_unchecked(size, align))
}

#[no_mangle]
pub unsafe fn __rust_dealloc(ptr: *mut u8, size: usize, align: usize) {
    GLOBAL_ALLOCATOR.dealloc(ptr, Layout::from_size_align_unchecked(size, align))
}

#[no_mangle]
pub unsafe fn __rust_realloc(ptr: *mut u8,
                             old_size: usize,
                             align: usize,
                             new_size: usize
) -> *mut u8 {
    GLOBAL_ALLOCATOR.realloc(ptr, Layout::from_size_align_unchecked(old_size, align), new_size)
}

#[no_mangle]
pub unsafe fn __rust_alloc_zeroed(size: usize, align: usize) -> *mut u8 {
    GLOBAL_ALLOCATOR.alloc_zeroed(Layout::from_size_align_unchecked(size, align))
}

[pacmancoder]

Found some interesting behavior. I was working on the following code snippet(part of the xTaskCreate wrapper wtih rust closures):

struct CallbackHolder<F: FnOnce()> {
    pub callback: Option<F>,
}

// Proxy function for closure calls
unsafe extern "C" fn rust_idf_freertos_task_callback_wrapper<F : FnOnce()>(data: *mut CVoid) {
    let holder = data as *mut MaybeUninit<CallbackHolder<F>>;

    (*(&mut *holder).as_mut_ptr()).callback.take().unwrap()();

    drop_in_place((&mut *holder).as_mut_ptr());
    free(holder as *mut CVoid);
}

It compiles without any problems, but when i tried to move the code to iram using #[link_section = ".iram1"] attribute I have faced the same issue as with dynamic allocation:

../../target/xtensa-esp8266-none-elf/release/libfirmware.a(firmware-5b7e9638ff0ed6ee.firmware.bstibb19-cgu.0.rcgu.o): In function `idf_hal::freertos::rust_idf_freertos_task_callback_wrapper::habb52adc3c00b90f':
firmware.bstibb19-cgu.0:(.iram1+0x16): dangerous relocation: l32r: literal placed after use: (.literal.literal+0x8)
firmware.bstibb19-cgu.0:(.iram1+0x1c): dangerous relocation: l32r: literal placed after use: (.literal.literal+0xc)
collect2: error: ld returned 1 exit status

[lexxvir]

@pacmancoder

Regarding first issue: how do you build libcore, liballoc and other crates from Rust std? Probably these crates don't get -Wa,--auto-lit-pools option when linking into object files.

Regarding second issue: I advise you to check freertos_rs crate, I use it with ESP32 without issues.

[pacmancoder]

@lexxvir I tried to substitute the original xtensa-lx106-elf-gcc binary with passthrough script for linker args, which I guess, should force any libraries including libcore and liballoc to build with -Wa,--auto-lit-pools - but still, no luck. 😞

Thanks for pointing out to freertos_rs crate, will definitely try it. 👍

[lexxvir]

@pacmancoder Do you have minimal example that reproduces the issue?

We already had issues with heap allocator stuff previously, see MabezDev/xtensa-rust-quickstart#2.
It seems that rustc somehow bypassing regular compilation chain when generates allocator code.
Maybe it worth to try to pass -Wa,--auto-lit-pools option at rustc itself compilation stage?

[pacmancoder]

@lexxvir Good, I'll prepare an example somewhere on the weekends

[bjoernQ]

@pacmancoder seems I get same problems you see as soon as I use a custom allocator and also no luck solving this currently

regarding your ugly workaround: how do you get this working without the linker complaining about duplicate symbols? Or did you patch the Rust sources to suppress generation of the __rust_* functions?

[pacmancoder]

@bjoernQ Hmm, somehow I didn't get any complaints about duplicate symbols. Note that on the rust side I am only compiling the static library, not the binary. So final symbol resoulution is up to IDF link process in my case. (ESP8266 RTOS SDK)

[bjoernQ]

@pacmancoder ah sure - that should explain it - I somehow missed the fact you are not building a final binary but a library. For me it should then work to patch Rust to not generate the __rust_* functions. Thanks for the hint!

[bjoernQ]

I looked a bit more into it and while I haven't found a solution I made at least some interesting observations:

After the build failed at the linking step I looked into target\xtensa-esp32-none-elf\release\deps - there are two "compilation units" - one is what was generated from my main.rs the other one seems to be what src/librustc_codegen_llvm/allocator.rs generates. The strange thing: for every other .BC file there is an .s file containing the assembly source but it's not there for the allocator stuff. Since in my understanding the Xtensa LLVM backend can only generate assembly source that's a bit strange. However there is an .o file which looked like this when inspected with nm:

__rg_alloc U
__rg_alloc_zeroed U
__rg_dealloc U
__rg_realloc U
.LCPI0_0 t 00000000
.literal t 00000000
.text.__rust_alloc t 00000000
.text.__rust_alloc_zeroed t 00000000
.text.__rust_dealloc t 00000000
.text.__rust_realloc t 00000000
__rust_alloc T 00000000 0000001d
__rust_alloc_zeroed T 00000000 0000001d
__rust_dealloc T 00000000 00000021
__rust_realloc T 00000000 00000029
10o1p4rgw0v8o449 a 00000000
.LCPI1_0 t 00000004
.LCPI2_0 t 00000008
.LCPI3_0 t 0000000c

I used llc to generate the .s file for the .bc ... that contains .cfi-lines which is odd since if I do the same for other .bc files there it's not there. I removed those .cfi lines and created an .o file from it via gcc ... now the nm looks like this:

__rg_alloc U
__rg_alloc_zeroed U
__rg_dealloc U
__rg_realloc U
.bss b 00000000
.data d 00000000
.literal t 00000000
.literal.literal t 00000000
.text t 00000000
.xt.lit n 00000000
.xt.prop n 00000000
.xtensa.info n 00000000
__rust_alloc T 00000000 00000011
10o1p4rgw0v8o449 a 00000000
Lfunc_end0 t 00000011
__rust_dealloc T 00000014 00000012
Lfunc_end1 t 00000026
__rust_realloc T 00000028 00000015
Lfunc_end2 t 0000003d
__rust_alloc_zeroed T 00000040 00000011
Lfunc_end3 t 00000051

Similar but different. And if I now re-run the failed linker invocation the linking succeeds. The resulting file is apparently wrong for some reason but at least I think this is an interesting thing. Maybe it can help someone else to solve it but at least it's an interesting thing I wanted to share.

[lkolbly]

So it seems Rust is indeed hand-generating the __rust_alloc et. al. symbols here. Building for ESP8266, the generated code compiles into:

$ xtensa-lx106-elf-objdump -d target/xtensa-esp8266-none-elf/release/deps/esp_app-1d743f34d9970dca.20hoktun4b60sa28.rcgu.o
Disassembly of section .literal:

00000000 <.LCPI0_0>:
   0:   000000                  ill
        ...

(etc. for LCPI1_0 through LCPI3_0)

Disassembly of section .text.__rust_alloc:

00000000 <__rust_alloc>:
   0:   f4c182                  addi    a8, a1, -12
   3:   081d                    mov.n   a1, a8
   5:   2109                    s32i.n  a0, a1, 8
   7:   038d                    mov.n   a8, a3
   9:   029d                    mov.n   a9, a2
   b:   0000a1                  l32r    a10, fffc000c <.LCPI3_0+0xfffc0000>
   e:   1199                    s32i.n  a9, a1, 4
  10:   0189                    s32i.n  a8, a1, 0
  12:   000ac0                  callx0  a10
  15:   2108                    l32i.n  a0, a1, 8
  17:   81cb                    addi.n  a8, a1, 12
  19:   081d                    mov.n   a1, a8
  1b:   f00d                    ret.n

With relocations that look like:

$ xtensa-lx106-elf-objdump -x target/xtensa-esp8266-none-elf/release/deps/esp_app-1d743f34d9970dca.20hoktun4b60sa28.rcgu.o
RELOCATION RECORDS FOR [.literal]:
OFFSET   TYPE              VALUE 
00000000 R_XTENSA_32       __rg_alloc
00000004 R_XTENSA_32       __rg_dealloc
00000008 R_XTENSA_32       __rg_realloc
0000000c R_XTENSA_32       __rg_alloc_zeroed


RELOCATION RECORDS FOR [.text.__rust_alloc]:
OFFSET   TYPE              VALUE 
0000000b R_XTENSA_SLOT0_OP  .literal

I interpret this to mean that ultimately, line 0xb (the l32r instruction) will reference .LCPI0_0, which in turn will be populated with the address of __rg_alloc. So, __rust_alloc is pretty much a shim which loads that address into a10 (line 0xb) and calls it (line 0x12).

Line 0xb is the one throwing the error, meaning that .LCPI0_0 is not being located properly in relation to __rust_alloc. I can't convince ld to tell me where it's trying to put things in this situation, sadly.

Interestingly, if I go the direction that @bjoernQ went and manually recompile the generated .bc file into a .o, __rust_alloc is generated with slightly different assembly:

$ llc target/xtensa-esp8266-none-elf/release/deps/esp_app-1d743f34d9970dca.20hoktun4b60sa28.rcgu.bc -o target/xtensa-esp8266-none-elf/release/deps/esp_app-1d743f34d9970dca.20hoktun4b60sa28.rcgu.o -filetype=obj
00000000 <__rust_alloc>:
   0:   004136                  entry   a1, 32
   3:   000081                  l32r    a8, fffc0004 <__rust_alloc_zeroed+0xfffbffc4>
   6:   02ad                    mov.n   a10, a2
   8:   03bd                    mov.n   a11, a3
   a:   0008e0                  callx8  a8
   d:   0a2d                    mov.n   a2, a10
   f:   f01d                    retw.n
  11:   0020f0                  nop

The l32r command has an offset of 0x3 here, whereas it used to have an offset of 0xb.

However, after converting the .bc to human-readable form and messing around with it until I got this ugly madness:

define i8* @__rust_alloc(i32, i32) {
  %3 = add i32 %0, 123
  %4 = mul i32 %0, %3
  %5 = tail call i8* @__rg_alloc(i32 %0, i32 %4)
  ret i8* %5
}

which compiles to this:

00000000 <__rust_alloc>:
   0:   005136                  entry   a1, 40
   3:   028d                    mov.n   a8, a2
   5:   7bc292                  addi    a9, a2, 123
   8:   82b290                  mull    a11, a2, a9
   b:   000091                  l32r    a9, fffc000c <__rust_alloc_zeroed+0xfffbffa0>
                        b: R_XTENSA_SLOT0_OP    .literal
   e:   02ad                    mov.n   a10, a2
  10:   0189                    s32i.n  a8, a1, 0
  12:   0009e0                  callx8  a9
  15:   0a2d                    mov.n   a2, a10
  17:   f01d                    retw.n
  19:   0020f0                  nop

Also works fine, so it doesn't appear to be a l32r alignment issue.

Comparing the sections of the two object files, the broken object file:

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000000  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .literal      00000010  00000000  00000000  00000034  2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .text.__rust_alloc 0000001d  00000000  00000000  00000044  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  3 .text.__rust_dealloc 00000022  00000000  00000000  00000064  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  4 .text.__rust_realloc 00000026  00000000  00000000  00000088  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  5 .text.__rust_alloc_zeroed 0000001d  00000000  00000000  000000b0  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  6 .eh_frame     00000078  00000000  00000000  000000d0  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA

has separate sections for each function, and the functions come after the literals. Compare to the working object file:

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000089  00000000  00000000  00000034  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .literal      00000010  00000000  00000000  000000bd  2**0
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  2 .eh_frame     00000068  00000000  00000000  000000d0  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA

where the literals start at 0xbd, after the text at 0x34.

I wouldn't normally think that this would matter at all, except that as far as I can tell the error message comes from code that looks like this.

self_address in that code appears to be the program counter in the ELF (VMA + offset + address, for example 0x4F in the bad object file. relocation appears to be... I dunno. Sometimes it's the "addend", sometimes it's the dest_addr for the relocation. Perhaps it's related to this hack?

I dunno. I have more investigation to do, for grins I'm building binutils locally and I'll add some tracing, but I have to go to bed so I'll post this for now in case someone else has an eureka moment.

TL;DR: I'm having the same problem, tried to root-cause but didn't quite grok enough to find the perfect solution. There's a workaround here which works great for me.

[lkolbly]

Alright, managed to compile a copy of ld which has some handy print statements in it, and got this:

asection.output_offset=11736 asection.size=29 self_address=1075916275 relocation=1074791708
/home/project/target/xtensa-esp8266-none-elf/release/deps/esp_app-1d743f34d9970dca.20hoktun4b60sa28.
rcgu.o: In function `__rust_alloc':
20hoktun4b60sa28:(.text.__rust_alloc+0xb): dangerous relocation: l32r: literal target out of range (
try using text-section-literals): .literal

i.e. self_address = 0x40212DF3 and relocation = 0x4010051C, my interpretation (I may be wrong) is that the former is the program counter and the latter is the target of the relocation operation, i.e. the address of the literal value storing the address of __rg_alloc.

An astute observer will notice that __rust_alloc is being placed in the instruction flash, whereas the literal is being placed 1,124,567 addresses away in the instruction RAM. l32r certainly can't encode that, so the question is why the difference? (and looking at a correctly generated binary, both get placed in iram in the 0x401... block)

So, as previously noted, the bad code generates __rust_alloc and friends as separate sections - .text.__rust_alloc, etc., whereas the good code places them all in the .text section. The linker script I'm using (from the IDF-style ESP8266 SDK) has a section like:

  /* Send .iram0 code to iram */
  .iram0.vectors :
  {
    _iram_start = ABSOLUTE(.);
   ...
    *(.text .literal)

Adding this:

    *(.text.__rust_alloc)
    *(.text.__rust_dealloc)
    *(.text.__rust_realloc)
    *(.text.__rust_alloc_zeroed)

in there to force those functions into the iram sections causes the build to pass as well.

Looking around that the code that generates the functions, I can't see why each function would get its own linker section. However, looking through a compiled app, it seems as though every rust function gets its own section, so it may be something intrinsic to how Rust compiles.

There are other functions which use l32r instructions in a similar way, in the various SDK libraries. However, for these, the literal value also gets its own section and so doesn't get plopped into iram away from the referencing function.

I don't know what the correct fix is (until further notice, I think I'll update the linker file for my own projects), but I think that's what's going on.

[lkolbly]

Just tested with the latest Xtensa LLVM from https://github.com/espressif/llvm-project, and it seems to be fixed just by using that.

[MabezDev]

Ah, I was about to suggest trying the new fork. The original is pretty broken in many ways.