QEMU 6.1.0 regression with CONFIG_THUMB2_KERNEL

[nathanchance]

Even with #325, upstream QEMU fails to boot. I have tracked it down to a commit in 6.1.0 and reported it upstream.

[nickdesaulniers]

ah, my version QEMU emulator version 5.2.0 (Debian 1:5.2+dfsg-11) was booting fine.

[nickdesaulniers]

[    0.080907] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000800
[    0.081365] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.0-rc1-00016-g4402e1c6209f #38
[    0.081771] Hardware name: Generic DT based system
[    0.082489] [<c030deb7>] (unwind_backtrace) from [<c0309921>] (show_stack+0xb/0xc)
[    0.083379] [<c0309921>] (show_stack) from [<c0cfc7d1>] (dump_stack_lvl+0x31/0x3a)
[    0.083657] [<c0cfc7d1>] (dump_stack_lvl) from [<c0cfc475>] (panic+0xc1/0x252)
[    0.083934] [<c0cfc475>] (panic) from [<c033e48d>] (do_exit+0x5e5/0x628)
[    0.084194] [<c033e48d>] (do_exit) from [<c033e725>] (do_group_exit+0x1/0x7a)
[    0.084436] [<c033e725>] (do_group_exit) from [<00000051>] (0x51)
[    0.085394] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000800 ]---

[nathanchance]

Richard Henderson replied that this appears to be a real alignment issue: https://lore.kernel.org/r/0699da7b-354f-aecc-a62f-e25693209af4@linaro.org/

[nickdesaulniers]

I can't find where it's coming from though. The disassembly from within GDB of your image doesn't match anything I built with debug info.

[ardbiesheuvel]

Could you share a bit more context? What config are you booting and how? And more context from the kernel log perhaps?

[nathanchance]

@ardbiesheuvel the config is multi_v7_defconfig + CONFIG_THUMB2_KERNEL=y in QEMU and the command line can be viewed here.

Nick should be able to give more log context.

[ardbiesheuvel]

I can confirm that your zImage-llvm doesn't boot with recent QEMU, but if I build mainline myself (using multi_v7_defconfig + CONFIG_THUMB2_KERNEL), it boots happily. I am using 'Debian clang version 11.0.1-2' with GNU binutils

[nathanchance]

Aha, the integrated assembler is the key to reproducing this. LLVM=1 LLVM_IAS=0 is fine, LLVM=1 LLVM_IAS=1 is not. Unfortunately, there are quite a few integrated assembler fixes in the 13.0.0 release so this configuration cannot be built with anything prior (there is apt.llvm.org or tuxmake for getting access to newer versions without too much hassle).

[nickdesaulniers]

Ok, some thoughts on IAS:

1. I'm curious if we add -no-integrated-as to just KBUILD_CFLAGS and NOT KBUILD_AFLAGS, if that makes a difference.
   2a. assuming it does, we might be able to bisect object files (C sources) with subdir-ccflags-y += -fno-integrated-as in various Makefiles.
   2b. assuming it does not, we might be able to bisect object files (ASM sources) with AS = arm-linux-gnueabi-as (hardcoded) in various Makefiles, confirming with make ... V=1 to check GAS is being used.

If we can narrow it down to an object file, I suspect we're not aligning a function or stack allocated variable correctly, and I wouldn't be surprised if interworking came into play here.

[nickdesaulniers]

Full kernel logs via qemu emulation: https://gist.github.com/nickdesaulniers/d184023431a7e5669bd1253c41509667

[ardbiesheuvel]

In the linked upstream report I see

0xc13038e2: e890 008c ldm.w r0, {r2, r3, r7}

so it is not a stack access. It also seems to me that we should be able to figure out which function covers this address, no?

I also see

R00=c13077ca

so this looks like it might be a literal pool that gets emitted at the wrong alignment? That would explain a lot actually, given the fact that it only happens with the integrated assembler, and only when building for Thumb2

[nathanchance]

I have at least narrowed it down to arch/arm/mm/tlb-v7.S, as the following diff allows me to boot with LLVM_IAS=1:

diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile
index 3510503bc5e6..43eb8911e592 100644
--- a/arch/arm/mm/Makefile
+++ b/arch/arm/mm/Makefile
@@ -74,6 +74,10 @@ obj-$(CONFIG_CPU_TLB_FA)     += tlb-fa.o

 AFLAGS_tlb-v6.o                :=-Wa,-march=armv6
 AFLAGS_tlb-v7.o                :=-Wa,-march=armv7-a
+ifdef CONFIG_AS_IS_LLVM
+AFLAGS_REMOVE_tlb-v7.o := -fintegrated-as
+AFLAGS_tlb-v7.o                += -fno-integrated-as --prefix=$(dir $(shell which $(CROSS_COMPILE)elfedit))$(notdir $(CROSS_COMPILE))
+endif

 obj-$(CONFIG_CPU_ARM7TDMI)     += proc-arm7tdmi.o
 obj-$(CONFIG_CPU_ARM720T)      += proc-arm720.o

With GNU as (LLVM_IAS=0):

$ llvm-objdump -dr arch/arm/mm/tlb-v7.o

arch/arm/mm/tlb-v7.o:   file format elf32-littlearm

Disassembly of section .text:

00000000 <v7wbi_flush_user_tlb_range>:
       0: 13 6a         ldr     r3, [r2, #32]
       2: d3 f8 78 31   ldr.w   r3, [r3, #376]
       6: bf f3 4b 8f   dsb     ish
       a: 4f ea 10 30   lsr.w   r0, r0, #12
       e: 4f ea 11 31   lsr.w   r1, r1, #12
      12: 03 f0 ff 03   and     r3, r3, #255
      16: 4f f0 00 03   mov.w   r3, #0
      1a: 43 ea 00 30   orr.w   r0, r3, r0, lsl #12
      1e: 4f ea 01 31   lsl.w   r1, r1, #12
      22: 08 ee 73 0f   mcr     p15, #0, r0, c8, c3, #3
      26: 00 f5 80 50   add.w   r0, r0, #4096
      2a: 88 42         cmp     r0, r1
      2c: f9 d3         blo     0x22 <v7wbi_flush_user_tlb_range+0x22> @ imm = #-14
      2e: bf f3 4b 8f   dsb     ish
      32: 70 47         bx      lr

00000034 <v7wbi_flush_kern_tlb_range>:
      34: bf f3 4b 8f   dsb     ish
      38: 4f ea 10 30   lsr.w   r0, r0, #12
      3c: 4f ea 11 31   lsr.w   r1, r1, #12
      40: 4f ea 00 30   lsl.w   r0, r0, #12
      44: 4f ea 01 31   lsl.w   r1, r1, #12
      48: 08 ee 73 0f   mcr     p15, #0, r0, c8, c3, #3
      4c: 00 f5 80 50   add.w   r0, r0, #4096
      50: 88 42         cmp     r0, r1
      52: f9 d3         blo     0x48 <v7wbi_flush_kern_tlb_range+0x14> @ imm = #-14
      54: bf f3 4b 8f   dsb     ish
      58: bf f3 6f 8f   isb     sy
      5c: 70 47         bx      lr
      5e: 00 bf         nop

Disassembly of section .init.text:

00000000 <v7wbi_tlb_fns>:
       0: 00 00 00 00 00 00 00 00         ........
       8: 00 00 f0 90                     ....

With the integrated assembler (LLVM_IAS=1):

$ llvm-objdump -dr arch/arm/mm/tlb-v7.o

arch/arm/mm/tlb-v7.o:   file format elf32-littlearm

Disassembly of section .text:

00000000 <v7wbi_flush_user_tlb_range>:
       0: 13 6a         ldr     r3, [r2, #32]
       2: d3 f8 78 31   ldr.w   r3, [r3, #376]
       6: bf f3 4b 8f   dsb     ish
       a: 4f ea 10 30   lsr.w   r0, r0, #12
       e: 4f ea 11 31   lsr.w   r1, r1, #12
      12: 03 f0 ff 03   and     r3, r3, #255

00000016 <.Ltmp0>:
      16: 4f f0 00 03   mov.w   r3, #0
      1a: 43 ea 00 30   orr.w   r0, r3, r0, lsl #12
      1e: 4f ea 01 31   lsl.w   r1, r1, #12

00000022 <.Ltmp3>:
      22: 08 ee 73 0f   mcr     p15, #0, r0, c8, c3, #3
      26: 00 f5 80 50   add.w   r0, r0, #4096
      2a: 88 42         cmp     r0, r1
      2c: f9 d3         blo     0x22 <.Ltmp3>           @ imm = #-14
      2e: bf f3 4b 8f   dsb     ish
      32: 70 47         bx      lr

00000034 <v7wbi_flush_kern_tlb_range>:
      34: bf f3 4b 8f   dsb     ish
      38: 4f ea 10 30   lsr.w   r0, r0, #12
      3c: 4f ea 11 31   lsr.w   r1, r1, #12
      40: 4f ea 00 30   lsl.w   r0, r0, #12
      44: 4f ea 01 31   lsl.w   r1, r1, #12

00000048 <.Ltmp6>:
      48: 08 ee 73 0f   mcr     p15, #0, r0, c8, c3, #3
      4c: 00 f5 80 50   add.w   r0, r0, #4096
      50: 88 42         cmp     r0, r1
      52: f9 d3         blo     0x48 <.Ltmp6>           @ imm = #-14
      54: bf f3 4b 8f   dsb     ish
      58: bf f3 6f 8f   isb     sy
      5c: 70 47         bx      lr

Disassembly of section .init.text:

00000000 <v7wbi_tlb_fns>:
       0: 00 00 00 00 00 00 00 00         ........

00000008 <.Ltmp8>:
       8: 00 00         movs    r0, r0
                        00000000:  R_ARM_ABS32  v7wbi_flush_user_tlb_range
                        00000004:  R_ARM_ABS32  v7wbi_flush_kern_tlb_range
       a: f0 90         str     r0, [sp, #960]