[RISCV64] ld.lld: error: relaxation not converged

[appujee]

The aosp toolchain is based on fc57f88

Steps to repro

Repro uploaded to google drive, can be downloaded from here Repro.tar

$ tar -xf repro.tar
$ cd repro
$ lld @response.txt

One can also this by building a full aosp

clone aosp-main-with-phones
$ source build/envsetup.sh
$ lunch aosp_cf_riscv64_phone-trunk_staging-userdebug 
$ m m net_test_stack

FAILED: out/soong/.intermediates/packages/modules/Bluetooth/system/stack/net_test_stack/android_riscv64_cfi/unstripped/net_test_stack64
prebuilts/clang/host/linux-x86/clang-r536225/bin/clang++ out/soong/.intermediates/bionic/libc/crtbegin_dynamic/android_riscv64/crtbegin_dynamic.o @out/soong/.intermediates/packages/modules/Bluetooth/system/stack/net_test_stack/android_riscv64_cfi/unstripped/net_test_stack64.rsp out/soong/.intermediates/bionic/libc/crtend_android/android_riscv64/crtend_android.o -o out/soong/.intermediates/packages/modules/Bluetooth/system/stack/net_test_stack/android_riscv64_cfi/unstripped/net_test_stack64 -target riscv64-linux-android10000 -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--build-id=md5 -Wl,--fatal-warnings -Wl,--no-undefined-version -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_stripped.a -Wl,--exclude-libs,libunwind_llvm.a -Wl,--exclude-libs,libunwind.a -fuse-ld=lld -Wl,--icf=safe -Wl,--no-demangle -Wl,--compress-debug-sections=zstd -Wl,--pack-dyn-relocs=android+relr -Wl,--no-undefined -march=rv64gcv_zba_zbb_zbs -Wl,-mllvm -Wl,-jump-is-expensive=false -Wl,-z,max-page-size=4096   -pie -nostdlib -Bdynamic -Wl,--gc-sections -Wl,-z,nocopyreloc -Wl,-rpath,\$ORIGIN -flto -fsanitize-cfi-cross-dso -fsanitize=cfi -Wl,-plugin-opt,O1 -fsanitize=bounds,cfi -fno-sanitize-link-runtime -Wl,--exclude-libs=libclang_rt.builtins-riscv64-android.a -Wl,--exclude-libs=libclang_rt.ubsan_minimal-riscv64-android.a -Wl,-dynamic-linker,/system/bin/linker64

[appujee]

cc: @MaskRay @topperc

[pirama-arumuga-nainar]

@topperc Did you get a chance to look into this?

[topperc]

@pirama-arumuga-nainar @hiraditya I'm having trouble extracting the reproducer.

[kito-cheng]

I also can't reproduce either...seems like Repro.tar has something wrong:

[kitoc@hsinchu18 llvm-123248]$ tar xvf repro.tar
repro/response.txt
repro/version.txt
repro/usr/local/google/home/appujee/g/aosp-main-with-phones/out/soong/.intermediates/bionic/libc/crtbegin_dynamic/android_riscv64/crtbegin_dynamic.o
repro/usr/local/google/home/appujee/g/aosp-main-with-phones/out/soong/.intermediates/packages/modules/Bluetooth/system/stack/test/fuzzers/a2dp/codec/a2dp_codec_cfg_fuzz/android_riscv64_fuzzer/obj/packages/modules/Bluetooth/system/stack/test/fuzzers/a2dp/codec/fuzz_a2dp_codec_config.o
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

Also I can't reproduce that by building aosp, not sure if I have missing any step:

$ mkdir aosp
 $cd aosp/
$ repo init -u https://android.googlesource.com/platform/manifest
$ repo sync
$ source build/envsetup.sh
$ lunch aosp_cf_riscv64_phone-trunk_staging-userdebug 
$ m net_test_stack
...
[100% 7792/7792] Install: out/target/product/vsoc_riscv64/data/nativetest64/net_test_stack/net_test_stack64

#### build completed successfully (06:15 (mm:ss)) ####

[Sharjeel-Khan]

If you are using aosp, you need to make sure https://r.android.com/3499717 was not reverted as it removed the temporary fix of no linker relaxation (-Wl,--no-relax) from platform/packages/modules/Bluetooth/Android.bp. You can also use mmma packages/modules/Bluetooth rather than m net_test_stack. The relaxation error has some non-deterministic behavior as it goes away and come back across commits in AOSP.

[Sharjeel-Khan]

I consistently get the issue on aosp if you do the following:

repo sync
source build/envsetup.sh
lunch aosp_cf_riscv64_phone-trunk_staging-userdebug
cd packages/modules/Bluetooth/
git checkout c7f8ad75d6838ab5ba323fa84bcbd13a1d02776c
m net_test_stack

I'll try to get a new repro if someone is trying it out of aosp.

[Sharjeel-Khan]

Here is a new repro: repro.tar

[kito-cheng]

I can reproduce within aosp, but I can't reproduce that with my local build (sha1 is 3302bef), also I can't find the corresponding upstream sha1 from aosp's lld.

Could you provide the common sha1 between aosp's LLVM and upstream LLVM so that I can try to reproduce that with my lld build?

[kitoc@sw07 aosp]$ prebuilts/clang/host/linux-x86/clang-r547379/bin/ld.lld --version
LLD 20.0.0 (/mnt/disks/build-disk/src/android/llvm-r547379-release/out/llvm-project/llvm b718bcaf8c198c82f3021447d943401e3ab5bd54) (compatible with GNU linkers)

[Sharjeel-Khan]

AOSP's LLVM is based on f142f8afe21bceb00fb495468aa0b5043e98c419 (Aug 23, 2024 commit)

[topperc]

As Kito said, it does not reproduce with trunk. I ran it through git-bisect and arrived at 2def1c4. That patch shouldn't have affected lld though and may just have perturbed the output of the lto step enough to hide the lld issue.

[topperc]

I think we have several calls from one function section to a later function section. The algorithm kind of assumes that when we relax a call that will shift the later function section forward by the same number of bytes we removed. In this case there are some sections with a 32 byte alignment between the call and call target. Maintaining the alignment for that section means that the call and call target sections aren't always the same distance apart. I think we need to guard band the relaxation by the maximum alignment of all sections between the call and the call target.

[topperc]

Ping @MaskRay

[topperc]

I think something like this will stop the relaxation loop and take the last state.

diff --git a/lld/ELF/Arch/RISCV.cpp b/lld/ELF/Arch/RISCV.cpp
index 4d8989a21b50..7e864574ccea 100644
--- a/lld/ELF/Arch/RISCV.cpp
+++ b/lld/ELF/Arch/RISCV.cpp
@@ -902,6 +902,9 @@ bool RISCV::relaxOnce(int pass) const {
   if (ctx.arg.relocatable)
     return false;
 
+  if (pass == 29)
+    return false;
+
   if (pass == 0)
     initSymbolAnchors(ctx);