[X86] Large stack frames are miscompiled with -fstack-clash-protection

[mconst]

On x86-64 with -fstack-clash-protection, the following code compiles to an infinite loop that consumes all physical memory:

void foo() {
    char x[0xa0000000];
}

It works fine on other architectures (including 32-bit x86), and it works without -fstack-clash-protection.

Here's the incorrect stack probe loop it produces:

        mov     r11, rsp
        sub     r11, -1610616832
.LBB0_1:
        sub     rsp, 4096
        mov     qword ptr [rsp], 0
        cmp     rsp, r11
        jne     .LBB0_1

(Note that if you look at LLVM's assembly output, it displays the second instruction misleadingly as sub r11, 2684350464, which makes it look correct. But in fact that's not a valid instruction; both llvm-mc and GNU as correctly refuse to assemble it. 64-bit SUB takes a 32-bit sign-extended immediate, so it's not possible to encode 2684350464.)

I've got a fix, which I'll submit shortly.

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: None (mconst)

On x86-64 with `-fstack-clash-protection`, the following code compiles to an infinite loop that consumes all physical memory:

void foo() {
    char x[0xa0000000];
}

It works fine on other architectures (including 32-bit x86), and it works without -fstack-clash-protection.

Here's the incorrect stack probe loop it produces:

        mov     r11, rsp
        sub     r11, -1610616832
.LBB0_1:
        sub     rsp, 4096
        mov     qword ptr [rsp], 0
        cmp     rsp, r11
        jne     .LBB0_1

(Note that if you look at LLVM's assembly output, it displays the second instruction misleadingly as sub r11, 2684350464, which makes it look correct. But in fact that's not a valid instruction; both llvm-mc and GNU as correctly refuse to assemble it. 64-bit SUB takes a 32-bit sign-extended immediate, so it's not possible to encode 2684350464.)

I've got a fix, which I'll submit shortly.