frame-larger-than in arch/powerpc/xmon/xmon.c

[shenki]

clang-7 and clang-8 (8.0.0-svn345470-1~exp1+0~20181028082208.133~1.gbpf10f36) result in this build error when building the ppc64 kernel:

arch/powerpc/xmon/xmon.c:452:12: warning: stack frame size of 2576 bytes in function 'xmon_core'
      [-Wframe-larger-than=]
static int xmon_core(struct pt_regs *regs, int fromipi)

Note that this is an error, as arch/powerpc builds with -Werror turned on.

[shenki]

@rnav did some digging and found out why xmon_core ends up so large:

Looks like it's to do with xmon_printf()
clang seems to be saving addresses to those in .rodata.str1.1 section on the stack

            798: R_PPC64_TOC16_HA    .rodata.str1.1+0x134e
     79c:    80 07 41 fa     std     r18,1920(r1)
     7a0:    00 00 54 3a     addi    r18,r20,0
            7a0: R_PPC64_TOC16_LO    .rodata.str1.1+0x185d
     7a4:    00 00 80 3a     li      r20,0
     7a8:    30 03 81 f8     std     r4,816(r1)
     7ac:    00 00 82 3c     addis   r4,r2,0
            7ac: R_PPC64_TOC16_HA    .rodata.str1.1+0x136f
     7b0:    28 03 81 f8     std     r4,808(r1)
     7b4:    00 00 82 3c     addis   r4,r2,0
            7b4: R_PPC64_TOC16_HA    .rodata.str1.1+0x1390
     7b8:    20 03 81 f8     std     r4,800(r1)
     7bc:    00 00 82 3c     addis   r4,r2,0
            7bc: R_PPC64_TOC16_HA    .rodata.str1.1+0x13b1
     7c0:    78 07 41 fa     std     r18,1912(r1)
     7c4:    18 03 81 f8     std     r4,792(r1)
     7c8:    00 00 82 3c     addis   r4,r2,0
            7c8: R_PPC64_TOC16_HA    .rodata.str1.1+0x13d2
     7cc:    10 03 81 f8     std     r4,784(r1)
     7d0:    00 00 82 3c     addis   r4,r2,0
            7d0: R_PPC64_TOC16_HA    .rodata.str1.1+0x13f3
     7d4:    08 03 81 f8     std     r4,776(r1)
     7d8:    00 00 82 3c     addis   r4,r2,0
            7d8: R_PPC64_TOC16_HA    .rodata.str1.1+0x1423
     7dc:    88 07 61 f8     std     r3,1928(r1)
     7e0:    00 00 7d 38     addi    r3,r29,0

> objdump -s -j .rodata.str1.1 ./xmon.o

<snip>

 1320 20202074 72617020 3d202534 6c780a00     trap = %4lx..
 1330 64617220 3d20252e 31366c78 20202064  dar = %.16lx   d
 1340 73697372 203d2025 2e386c78 0a006d73  sisr = %.8lx..ms
 1350 72202020 203d2025 2e31366c 78202073  r    = %.16lx  s
 1360 70726730 203d2025 2e31366c 780a0070  prg0 = %.16lx..p
 1370 76722020 20203d20 252e3136 6c782020  vr    = %.16lx  
 1380 73707267 31203d20 252e3136 6c780a00  sprg1 = %.16lx..
 1390 64656320 2020203d 20252e31 366c7820  dec    = %.16lx 
 13a0 20737072 6732203d 20252e31 366c780a   sprg2 = %.16lx.
 13b0 00737020 20202020 3d20252e 31366c78  .sp     = %.16lx
 13c0 20207370 72673320 3d20252e 31366c78    sprg3 = %.16lx
 13d0 0a00746f 63202020 203d2025 2e31366c  ..toc    = %.16l
 13e0 78202064 61722020 203d2025 2e31366c  x  dar   = %.16l
 13f0 780a0073 72723020 20203d20 252e3136  x..srr0   = %.16
 1400 6c782020 73727231 20203d20 252e3136  lx  srr1  = %.16
 1410 6c782064 73697372 20203d20 252e386c  lx dsisr  = %.8l
 1420 780a0064 73637220 20203d20 252e3136  x..dscr   = %.16
 1430 6c782020 70707220 20203d20 252e3136  lx  ppr   = %.16
 1440 6c782070 69722020 20203d20 252e386c  lx pir    = %.8l
 1450 780a0061 6d722020 20203d20 252e3136  x..amr    = %.16
 1460 6c782020 75616d6f 72203d20 252e3136  lx  uamor = %.16

Clang:

       printf("Invalid number.\n");
    1ab0:    80 07 61 e8     ld      r3,1920(r1)
    1ab4:    01 00 00 48     bl      1ab4 <xmon_core+0x1a00>
            1ab4: R_PPC64_REL24    xmon_printf
    1ab8:    00 00 00 60     nop

GCC:

    printf("Invalid number.\n");
    6200:    00 00 62 3c     addis   r3,r2,0
            6200: R_PPC64_TOC16_HA    .rodata.str1.8+0x2420
    6204:    00 00 63 38     addi    r3,r3,0
            6204: R_PPC64_TOC16_LO    .rodata.str1.8+0x2420
    6208:    01 00 00 48     bl      6208 <xmon_core+0x1b98>
            6208: R_PPC64_REL24    xmon_printf
    620c:    00 00 00 60     nop

[shenki]

Kernel patch posted: https://patchwork.ozlabs.org/patch/991235/

This is a workaround. I think this should be fixed in clang though before we can consider this resolved.

[nickdesaulniers]

Still unable to repro.

llvm-11:

$ git revert 9c87156
<comment out building vdso code in arch/powerpc/kernel/vdso/Makefile>
$ make CROSS_COMPILE=powerpc64le-linux-gnu- ARCH=powerpc LLVM=1 LLVM_IAS=0 -j128 defconfig arch/powerpc/xmon/xmon.o
$

llvm-18:

$ git revert 9c87156
$ make ARCH=powerpc LLVM=1 -j128 defconfig arch/powerpc/xmon/xmon.o

I'm also unable to reproduce the reported disassembly with clang-11+. Even tried on godbolt: https://godbolt.org/z/bqhjoWsTa

[nickdesaulniers]

Sent: https://lore.kernel.org/llvm/20230817-ppc_xmon-v1-1-8cc2d51b9995@google.com/

[nickdesaulniers]

accepted: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id=7f3c5d099b6f8452dc4dcfe4179ea48e6a13d0eb

[nathanchance]

Unfortunately, it seems that there was some poor code generation with LLVM 12 and 13, as reported by CI (11 and 14 appear clean).

LLVM 12:

arch/powerpc/xmon/xmon.c:489:12: error: stack frame size of 2064 bytes in function 'xmon_core' [-Werror,-Wframe-larger-than=]
static int xmon_core(struct pt_regs *regs, volatile int fromipi)
           ^
1 error generated.

LLVM 13:

arch/powerpc/xmon/xmon.c:489:12: error: stack frame size (2096) exceeds limit (2048) in function 'xmon_core' [-Werror,-Wframe-larger-than]
static int xmon_core(struct pt_regs *regs, volatile int fromipi)
           ^
1 error generated.

Realistically, I think we can just ignore this and enable CONFIG_PPC_DISABLE_WERROR=y in CI for powernv_defconfig or maybe send something like

diff --git a/arch/powerpc/Kconfig.debug b/arch/powerpc/Kconfig.debug
index 2a54fadbeaf5..cb8e3f92bcf6 100644
--- a/arch/powerpc/Kconfig.debug
+++ b/arch/powerpc/Kconfig.debug
@@ -2,6 +2,8 @@

 config PPC_DISABLE_WERROR
        bool "Don't build arch/powerpc code with -Werror"
+       # There are a few known warnings for clang versions older than 14.0.0
+       default y if CC_IS_CLANG && CLANG_VERSION < 140000
        help
          This option tells the compiler NOT to build the code under
          arch/powerpc with the -Werror flag (which means warnings

since we still see/workaround #1445 as well.

[nickdesaulniers]

kbuild test robot found a randconfig that breaks with clang-14: https://lore.kernel.org/llvm/202308260344.Vc4Giuk7-lkp@intel.com/

[nickdesaulniers]

Sent a revert for this: https://lore.kernel.org/llvm/20230828-ppc_rerevert-v2-1-46b71a3656c6@google.com/

[nickdesaulniers]

clang-13:

$ git revert 90bae4d99beb1f31d8bde7
$ make CROSS_COMPILE=powerpc64le-linux-gnu- ARCH=powerpc LLVM=1 LLVM_IAS=0 -j128 defconfig arch/powerpc/xmon/xmon.o
arch/powerpc/xmon/xmon.c:490:12: error: stack frame size (2080) exceeds limit (2048) in function 'xmon_core' [-Werror,-Wframe-larger-than]
static int xmon_core(struct pt_regs *regs, volatile int fromipi)
           ^

Looking at xmon_core in LLVM IR, these look sus:

%struct.arch_hw_breakpoint = type { i64, i16, i16, i16, i8, i8 }                
...
define internal fastcc signext i32 @xmon_core(%struct.pt_regs* %regs, i32 signext %fromipi) unnamed_addr #0 {
...
  %null_brk.i.i568 = alloca %struct.arch_hw_breakpoint, align 8                 
...
%tmp.i.i = alloca [64 x i8], align 1
...
  %null_brk.i.i = alloca %struct.arch_hw_breakpoint, align 8        
...
 %recurse_jmp = alloca [23 x i64], align 8            

The two arch_hw_breakpoint do have lifetime markers, tmp does not. recurse_jmp has lifetime markers, but is effectively the entirety of the function. I wonder if the lifetime of recurse_jmp can be manually reduced (by moving its definition into a reduced scope?

In #39 (comment), I have a better idea of some of the issues in clang that are resulting in excessive stack usage, and how better to debug these kinds of issues in the future.

I think we'll leave this workaround in the kernel for now. I'll keep this issue open for now; once #39 is fixed, then it might be worthwhile to either:

• update the in kernel workaround to be and older only
• revert 90bae4d if we drop support for these older releases of clang with known stack size issues
• try moving recurse_jmp to a reduced scope.

I suspect the former suggestion will make it easier to know when precisely to do the latter. For now, I'll leave this open until #39 fixes are landed in clang, as I'd like to eliminate those sources first.