Error linking kernel/futex.o

[nathanchance]

I was going to resend the patch that fixes #287 but as I was trying to come up with some testing steps for people, I ran into this error with an arm allyesconfig build (little endian):

arm-linux-gnueabi-ld: kernel/futex.o:(.text.fixup+0x5c): relocation truncated to fit: R_ARM_JUMP24 against `.init.text'
make[2]: *** [/linux-next/Makefile:1021: vmlinux] Error 1
make[1]: *** [/linux-next/Makefile:296: __build_one_by_one] Error 2

I need the following patches to avoid other link/build errors:

• https://lore.kernel.org/lkml/20190122233749.42220-1-trong@android.com/
• https://lore.kernel.org/lkml/20181215212304.19390-1-natechancellor@gmail.com/
• https://raw.githubusercontent.com/nathanchance/patches/c313b2fa0efb/linux/build-hax/0003-DO-NOT-UPSTREAM-ARM-Don-t-select-HAVE_FUNCTION_TRACE.patch

It doesn't appear to be a kernel regression because it goes all the way back to v4.19 which I know I had building with ld.bfd but I switched to ld.lld a few weeks ago and have had no issues with linking since then.

Unfortunately, I don't have time today to try and see if LLVM or binutils is the issue (even with the new server I am renting with a Ryzen 7, it takes 15-20 minutes a build clean), hence opening this issue. I've been reproducing in our Docker container (to eliminate any environment related issues) with the following commands:

git clone -b arm-link-fail-v5.0 git://github.com/nathanchance/linux

cd linux

docker run --interactive --mount type=bind,source="${PWD}",target=/"$(basename "${PWD}")",readonly --rm --tty --workdir /"$(basename "${PWD}")" clangbuiltlinux/ubuntu

cat <<EOF > /build.sh
rm -rf /out && mkdir -p /out

echo "CONFIG_CPU_BIG_ENDIAN=n" >> /out/le.config

time make -j"$(nproc --all)" ARCH=arm CC="ccache clang-9" HOSTCC="ccache clang-9" CROSS_COMPILE=arm-linux-gnueabi- KCONFIG_ALLCONFIG=/out/le.config O=/out allyesconfig vmlinux
EOF

bash /build.sh

binutils has never been upgraded in our Docker container so it seems unlikely that it is the cause of the regression, making it look like Clang is probably the issue. Will keep the issue updated with my findings.

[nickdesaulniers]

cc @smithp35

…

On Wed, Jan 23, 2019, 10:40 AM Nathan Chancellor ***@***.*** wrote: I was going to resend the patch that fixes #287 <#287> but as I was trying to come up with some testing steps for people, I ran into this error with an arm allyesconfig build (little endian): arm-linux-gnueabi-ld: kernel/futex.o:(.text.fixup+0x5c): relocation truncated to fit: R_ARM_JUMP24 against `.init.text' make[2]: *** [/linux-next/Makefile:1021: vmlinux] Error 1 make[1]: *** [/linux-next/Makefile:296: __build_one_by_one] Error 2 I need the following patches to avoid other link/build errors: - ***@***.***/ - ***@***.***/ - https://raw.githubusercontent.com/nathanchance/patches/c313b2fa0efb/linux/build-hax/0003-DO-NOT-UPSTREAM-ARM-Don-t-select-HAVE_FUNCTION_TRACE.patch It doesn't appear to be a kernel regression because it goes all the way back to v4.19 which I know I had building with ld.bfd but I switched to ld.lld a few weeks ago and have had no issues with linking since then. Unfortunately, I don't have time today to try and see if LLVM or binutils is the issue (even with the new server I am renting with a Ryzen 7, it takes 15-20 minutes a build clean), hence opening this issue. I've been reproducing in our Docker container (to eliminate any environment related issues) with the following commands: git clone -b arm-link-fail-v5.0 git://github.com/nathanchance/linux cd linux docker run --interactive --mount type=bind,source="${PWD}",target=/"$(basename "${PWD}")",readonly --rm --tty --workdir /"$(basename "${PWD}")" clangbuiltlinux/ubuntu cat <<EOF > /build.sh rm -rf /out && mkdir -p /out echo "CONFIG_CPU_BIG_ENDIAN=n" >> /out/le.config time make -j"$(nproc --all)" ARCH=arm CC="ccache clang-9" HOSTCC="ccache clang-9" CROSS_COMPILE=arm-linux-gnueabi- KCONFIG_ALLCONFIG=/out/le.config O=/out allyesconfig vmlinux EOF bash /build.sh binutils has never been upgraded in our Docker container so it seems unlikely that it is the cause of the regression, making it look like Clang is probably the issue. Will keep the issue updated with my findings. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#325>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvUX0DbFR60PJ5czPymtmOcwe1wmioEks5vGKyagaJpZM4aPZKK> .

[nickdesaulniers]

It doesn't appear to be a kernel regression because it goes all the way back to v4.19 which I know I had building with ld.bfd but I switched to ld.lld a few weeks ago and have had no issues with linking since then.

Were those allyesconfig's though? Maybe a new config got added that broke this?

[smithp35]

I can't see a smoking gun with either clang or gcc right now.

That particular error message is coming from a bit of inline assembly in futex.h
#define __futex_atomic_ex_table(err_reg)
"3:\n"
" .pushsection __ex_table,"a"\n"
" .align 3\n"
" .long 1b, 4f, 2b, 4f\n"
" .popsection\n"
" .pushsection .text.fixup,"ax"\n"
" .align 2\n"
"4: mov %0, " err_reg "\n"
" b 3b\n"
" .popsection"

clang (integrated-as) will define a .Ltmp0 symbol for 3: and hence the B 3b is represented as an R_ARM_JUMP24 to .Ltmp0. This seems fine to me and shouldn't be a problem for a linker to extend the range with a stub if .text.fixup is placed a long way away from .Ltmp0.

clang (-fno-integrated-as) and gcc (I'm using 5.4) encode 3: differently. They use the section symbol for .Ltmp0 + addend (where addend is encoded in the immediate field of the branch). This looks like it is handled correctly by the ld.bfd stub generation code. To be honest I don't think that LLD will handle this in all cases as a branch to a section symbol + offset is extremely rare, particularly in Arm as it does not support ARM/Thumb interworking.

From bitter experience a lot of stub generating bugs can be latent for years, it only takes the contents of a few sections to move a bit and something can go from being clearly out of range to only just out of range to trigger a subtle bug.

[smithp35]

I've raised https://llvm.org/pr40438 to cover LLD not including the addend when measuring range to branch + symbol + addend. The probability of a relocation out of range error is small as we would need source of relocation < (symbol + addend), and (symbol - source) < addend.

[nathanchance]

Were those allyesconfig's though?

Yes.

I realized that I have never actually tested mainline continuously (I only test -next) so I went back to next-20181210 because that's when I first started building arm allyesconfig (commit) and applied the patches listed in the first comment (plus a few others needed to fix errors along the way, all of which ended up in mainline) and sure enough, it builds and links successfully. I went ahead and started walking forward and was able to build successfully on next-20181224 but not on next-20190102 (the following next version since Stephen took the next week off). next-20181224 was based on v4.20-rc7 so I went ahead and started bisecting. As it turns out...

git bisect start '--term-new=fixed' '--term-old=unfixed'
# fixed: [6a1d293238c1774cef23c25b5188b0c134c29a34] Add linux-next specific files for 20181224
git bisect fixed 6a1d293238c1774cef23c25b5188b0c134c29a34
# unfixed: [7566ec393f4161572ba6f11ad5171fd5d59b0fbd] Linux 4.20-rc7
git bisect unfixed 7566ec393f4161572ba6f11ad5171fd5d59b0fbd
# fixed: [c8487f62082d0acc3faae807339650e9a5dd7ecf] Merge remote-tracking branch 'mtd/mtd/next'
git bisect fixed c8487f62082d0acc3faae807339650e9a5dd7ecf
# fixed: [2c04e516a0eab28f3027308c4e0576c6b0fe6196] Merge remote-tracking branch 'pstore/for-next/pstore'
git bisect fixed 2c04e516a0eab28f3027308c4e0576c6b0fe6196
# fixed: [cb604541bd60b80ab514d0de99d08b6f1f5a3218] Merge remote-tracking branch 'arm-soc/for-next'
git bisect fixed cb604541bd60b80ab514d0de99d08b6f1f5a3218
# unfixed: [e2e2dcf6f40b0777185c20ca84e6388923a55df8] ARM: Document merges
git bisect unfixed e2e2dcf6f40b0777185c20ca84e6388923a55df8
# unfixed: [48a8e49987d9a119dadd620b89da099a12624e3d] Merge remote-tracking branch 'hyperv-fixes/hyperv-fixes'
git bisect unfixed 48a8e49987d9a119dadd620b89da099a12624e3d
# fixed: [a902309d44036699130ba2b9a603db68cc0808a3] Merge remote-tracking branch 'arm/for-next'
git bisect fixed a902309d44036699130ba2b9a603db68cc0808a3
# fixed: [9cc5b4676d0c2ae34c15edba8e9060b8b08028ac] Merge remote-tracking branch 'leaks/leaks-next'
git bisect fixed 9cc5b4676d0c2ae34c15edba8e9060b8b08028ac
# unfixed: [11fda148040bb987949f9b957aceb483407e32cb] kbuild: remove $(obj)/ prefixes in ./Kbuild
git bisect unfixed 11fda148040bb987949f9b957aceb483407e32cb
# unfixed: [2f8967f21a18a8e520a9d17bde204e3b4f164724] kconfig: split the lexer out of zconf.y
git bisect unfixed 2f8967f21a18a8e520a9d17bde204e3b4f164724
# unfixed: [9f84a0f7f1a613335a9aaf0abd5858906752bf93] leaking_addresses: check if file name contains address
git bisect unfixed 9f84a0f7f1a613335a9aaf0abd5858906752bf93
# unfixed: [5cfc879caee810828d90aec808d85560f34f02af] pcmcia: remove per-arch PCMCIA config entry
git bisect unfixed 5cfc879caee810828d90aec808d85560f34f02af
# fixed: [5cd6239506cb97fb00c5992c1adaf6f9f788c5f2] kernel hacking: support building kernel with -Og optimization level
git bisect fixed 5cd6239506cb97fb00c5992c1adaf6f9f788c5f2
# unfixed: [892a5210d8977d1b805aaaceb548c0ff72329494] openrisc: make function cache_loop() inline
git bisect unfixed 892a5210d8977d1b805aaaceb548c0ff72329494
# fixed: [eb14ab6c93199e7cc0186482dba0769c01c06e35] ARM: mm: fix build error in fix_to_virt with -Og optimization level
git bisect fixed eb14ab6c93199e7cc0186482dba0769c01c06e35
# fixed: [13d2598cc964173bb063cffda9de778a89cdcfcd] kernel hacking: add a config option to disable compiler auto-inlining
git bisect fixed 13d2598cc964173bb063cffda9de778a89cdcfcd
# first fixed commit: [13d2598cc964173bb063cffda9de778a89cdcfcd] kernel hacking: add a config option to disable compiler auto-inlining

CONFIG_NO_AUTO_INLINE was "fixing" it and I never noticed because it was always there when building allyesconfig. However, it didn't make it into 5.0-rc1 then Masahiro dropped it from his tree.

If I mark futex_atomic_cmpxchg_inatomic as noinline like so, the kernel successfully links with ld.bfd:

diff --git a/arch/arm/include/asm/futex.h b/arch/arm/include/asm/futex.h
index 0a46676b4245..1d723584addc 100644
--- a/arch/arm/include/asm/futex.h
+++ b/arch/arm/include/asm/futex.h
@@ -42,7 +42,7 @@
        uaccess_restore(__ua_flags);                            \
 })

-static inline int
+static noinline int
 futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
                              u32 oldval, u32 newval)
 {

Note: ld.lld works just fine, even without that diff.

[nickdesaulniers]

MTK reports hitting this as well. I can repro, but it seems even with a pared down config, MTK hits it as well (seems I can repro with AOSP clang r353983c, but not newer builds of clang with that config).
config.arm.thumb2kernel.txt

[nickdesaulniers]

arm-linux-gnueabi-ld: kernel/futex.o:(.text.fixup+0x5c): relocation truncated to fit: R_ARM_JUMP24 against `.init.text'

When I build with Clang, I see:

$ llvm-objdump -Dr kernel/futex.o
...
Disassembly of section .text.fixup:
...
      58: 02 c0 a0 e1                   mov     r12, r2
      5c: 5f 00 00 ea                   b       #380 <$a+0x1e0>
                        0000005c:  R_ARM_JUMP24 .init.text
...

Which is the relocation that bfd is complaining about.

I suspect that the sections in between .init.text and .text.fixup are larger when built w/ Clang than w/ GCC (observed when linking .tmp_vmlinux1, ie. before arch/arm/boot/compressed/vmlinux.lds.S munges .text.fixup into .text). I'll bet moving a function that generated that relocation out of the init section allows for the relocation to fit, but it seems that kernel/futex.c is arch independent, so I'm not sure that's a viable solution.

clang (-fno-integrated-as) and gcc (I'm using 5.4) encode 3: differently.

Right, the R_ARM_JUMP24 becomes problematic when the sections are placed too far apart.

https://gcc.gnu.org/ml/gcc/2011-10/msg00224.html I think discusses a similar error, with the response being "sorry, I can't do that").

In arch/arm/kernel/vmlinux.lds.S, RO_DATA(PAGE_SIZE) is what defines all of the sections between .text and .ARM.unwind_idx, ie.

.rodata
.modinfo
.pci_fixup
__ksymtab
__ksymtab_gpl
__ksymtab_strings
__param
__modver
__ex_table

(.rodata is pretty big, it's the second largest section after .text).

So if we could move those (and maybe ARM_UNWIND_SECTIONS, NOTES, and ARM_VECTORS, then .text.fixup wouldn't be so far away from .init.text.

I'm thinking:

diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index 23150c0f0f4d..619cb62dd715 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -68,8 +68,6 @@ SECTIONS
 #endif
 	_etext = .;			/* End of text section */
 
-	RO_DATA(PAGE_SIZE)
-
 	. = ALIGN(4);
 	__ex_table : AT(ADDR(__ex_table) - LOAD_OFFSET) {
 		__start___ex_table = .;
@@ -77,12 +75,6 @@ SECTIONS
 		__stop___ex_table = .;
 	}
 
-#ifdef CONFIG_ARM_UNWIND
-	ARM_UNWIND_SECTIONS
-#endif
-
-	NOTES
-
 #ifdef CONFIG_STRICT_KERNEL_RWX
 	. = ALIGN(1<<SECTION_SHIFT);
 #else
@@ -142,6 +134,14 @@ SECTIONS
 #endif
 	__init_end = .;
 
+	RO_DATA(PAGE_SIZE)
+
+#ifdef CONFIG_ARM_UNWIND
+	ARM_UNWIND_SECTIONS
+#endif
+
+	NOTES
+
 	_sdata = .;
 	RW_DATA_SECTION(L1_CACHE_BYTES, PAGE_SIZE, THREAD_SIZE)
 	_edata = .;

Strictly speaking, just the move of RO_DATA might be enough. We might also want to look at other arch's too see if they place .rodata here or somewhere else. Without knowing what kind of data is encoded in all of these sections, moving any of them may trigger other failures. Untested.

[nickdesaulniers]

(doesn't work) 💩

[nickdesaulniers]

@nathanchance sent a reference to this thread we had with @arndb and @ardbiesheuvel .

Probably should follow up on @ardbiesheuvel 's suggestion: https://lore.kernel.org/lkml/CAKv+Gu__jgnm00ou+3FsqRBZWeDFy5Qi+1eiXeMKqc+hVSXomg@mail.gmail.com/#t