clang-repl value unit test fails on 32 bit Arm

[DavidSpickett]

Since #89811 landed one of the unit tests has been failing on Arm 32 bit:

******************** TEST 'Clang-Unit :: Interpreter/./ClangReplInterpreterTests/10/26' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/unittests/Interpreter/./ClangReplInterpreterTests-Clang-Unit-2678467-10-26.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=26 GTEST_SHARD_INDEX=10 /home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/unittests/Interpreter/./ClangReplInterpreterTests
--
Script:
--
/home/tcwg-buildbot/worker/clang-armv8-quick/stage1/tools/clang/unittests/Interpreter/./ClangReplInterpreterTests --gtest_filter=InterpreterTest.Value
--
../llvm/clang/unittests/Interpreter/InterpreterTest.cpp:293: Failure
Expected equality of these values:
  V1.getInt()
    Which is: 0
  42

It does not fail on AArch64. It does fail under qemu emulation too, but with different results. Which points to some UB in the program.

[DavidSpickett]

Compiling debug builds natively is a nightmare these days due to RAM constraints, so I cross compiled it using the instructions I wrote in #94741 (comment).

When run under qemu we get different numbers:

[ RUN      ] InterpreterTest.Value
/work/open_source/llvm-project/clang/unittests/Interpreter/InterpreterTest.cpp:98: Failure
Expected equality of these values:
  V1.getInt()
    Which is: -390410176
  42

/work/open_source/llvm-project/clang/unittests/Interpreter/InterpreterTest.cpp:99: Failure
Expected equality of these values:
  V1.convertTo<int>()
    Which is: -390410176
  42

Then I debugged it using qemu's GDB stub and gdb-multiarch:

$ gdb-multiarch /work/open_source/build-llvm-arm/tools/clang/unittests/Interpreter/ClangReplInterpreterTests -ex "set sysroot /work/open_source/arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/arm-none-linux-gnueabihf/libc/" -ex "set solib-search-path /work/open_source/build-llvm-arm/lib/" -ex "target remote :1234" -ex "b clang/lib/Interpreter/Interpreter.cpp:870" -ex "c"

(gdb) bt
#0  __clang_Interpreter_SetValueNoAlloc (This=0xa4b98, OutVal=0xa4bb8, OpaqueType=0xcfa70) at /work/open_source/llvm-project/clang/lib/Interpreter/Interpreter.cpp:872
#1  0xe8bad030 in __stmts__0 ()
#2  0xe8bad058 in _GLOBAL__sub_I_incr_module_3 ()
#3  0xe8bad048 in __orc_init_func.incr_module_3 ()
#4  0xf9f418d8 in (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize (this=0x141d40, JD=...) at /work/open_source/llvm-project/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp:240
#5  0xff715dd0 in llvm::orc::LLJIT::initialize (this=0xa3ca8, JD=...) at /work/open_source/llvm-project/llvm/include/llvm/ExecutionEngine/Orc/LLJIT.h:199
#6  0xff71389c in clang::IncrementalExecutor::runCtors (this=0x140398) at /work/open_source/llvm-project/clang/lib/Interpreter/IncrementalExecutor.cpp:98
#7  0xff73931c in clang::Interpreter::Execute (this=0xa4b98, T=...) at /work/open_source/llvm-project/clang/lib/Interpreter/Interpreter.cpp:450
#8  0xff739458 in clang::Interpreter::ParseAndExecute (this=0xa4b98, Code=..., V=0xfffef010) at /work/open_source/llvm-project/clang/lib/Interpreter/Interpreter.cpp:462
#9  0x0002d1c4 in (anonymous namespace)::InterpreterTest_Value_Test::TestBody (this=0x7da08) at /work/open_source/llvm-project/clang/unittests/Interpreter/InterpreterTest.cpp:95
#10 0xf8b7d9ac in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x7da08, method=&virtual testing::Test::TestBody(),
    location=0xf8b18b70 "the test body") at /work/open_source/llvm-project/third-party/unittest/googletest/src/gtest.cc:2612
#11 0xf8b71f30 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x7da08, method=&virtual testing::Test::TestBody(), location=0xf8b18b70 "the test body")
    at /work/open_source/llvm-project/third-party/unittest/googletest/src/gtest.cc:2667
#12 0xf8b49cac in testing::Test::Run (this=0x7da08) at /work/open_source/llvm-project/third-party/unittest/googletest/src/gtest.cc:2687
#13 0xf8b4a568 in testing::TestInfo::Run (this=0x68d08) at /work/open_source/llvm-project/third-party/unittest/googletest/src/gtest.cc:2836
#14 0xf8b4af68 in testing::TestSuite::Run (this=0x7fb40) at /work/open_source/llvm-project/third-party/unittest/googletest/src/gtest.cc:3015
<...>

What's interesting is that the value seen in the failure:

[ RUN      ] InterpreterTest.Value
/work/open_source/llvm-project/clang/unittests/Interpreter/InterpreterTest.cpp:98: Failure
Expected equality of these values:
  V1.getInt()
    Which is: -390410176
  42

Turns up in the varargs as the first argument:

(gdb) p *(int*)args.__ap
$5 = -390410176

Which suggests that the code in __clang_Interpreter_SetValueNoAlloc is doing what it's supposed to, but being given incorrect values.

If I go up the callstack and track the value of what would be the first vararg, r3, I eventually get back to:

(gdb) up
#1  0xe8bad030 in __stmts__0 ()
(gdb) p/x $r3
$5 = 0xe8bad040
(gdb) up
#2  0xe8bad058 in _GLOBAL__sub_I_incr_module_3 ()
(gdb) p/x $r3
$6 = 0xe8bad040
(gdb) up
#3  0xe8bad048 in __orc_init_func.incr_module_3 ()
(gdb) p/x $r3
$7 = 0xe8bad040
(gdb) up
#4  0xf9f418d8 in (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize (this=0x141d40, JD=...) at /work/open_source/llvm-project/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp:240
(gdb) p/x $r3
$8 = 0xe8bad040

Which is branching to the address of __orc_init_func.incr_module_3:

│   0xf9f418cc <(anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&)+540>  str     r0, [r11, #-20] ; 0xffffffec                                              │
│   0xf9f418d0 <(anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&)+544>  ldr     r3, [r11, #-20] ; 0xffffffec                                              │
│   0xf9f418d4 <(anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&)+548>  blx     r3                                                                        │
│  >0xf9f418d8 <(anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&)+552>  sub     r3, r11, #88    ; 0x58 

According to the ABI and compiler generated code, the value of the int would be in r3 (see https://godbolt.org/z/n6r1Tv9sP). So I think something between GenericLLVMIRPlatformSupport::initialize and __clang_Interpreter_SetValueNoAlloc should have setup the value of r3.

I checked this on AArch64 and saw that __stmts__0 does this.

We get the same set of function calls:

(lldb) bt
* thread #1, name = 'ClangReplInterp', stop reason = signal SIGTRAP
  * frame #0: 0x0000aaaaadd27e98 ClangReplInterpreterTests`__clang_Interpreter_SetValueNoAlloc(This=0x0000aaaabac90a60, OutVal=0x0000aaaabac90aa0, OpaqueType=0x0000aaaabacc72a0) at Interpreter.cpp:871:3
    frame #1: 0x0000fffff7ffa030 JIT(0xfffff7ff6000)`__stmts__0 + 48
    frame #2: 0x0000fffff7ffa05c JIT(0xfffff7ff6000)`_GLOBAL__sub_I_incr_module_3 + 12
    frame #3: 0x0000fffff7ffa048 JIT(0xfffff7ff6000)`__orc_init_func.incr_module_3 + 8
    frame #4: 0x0000aaaaacec3370 ClangReplInterpreterTests`(anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(this=0x0000aaaabac40300, JD=0x0000aaaabadd07d0) at LLJIT.cpp:240:9
    frame #5: 0x0000aaaaadd4011c ClangReplInterpreterTests`llvm::orc::LLJIT::initialize(this=0x0000aaaabac85190, JD=0x0000aaaabadd07d0) at LLJIT.h:199:16
    frame #6: 0x0000aaaaadd3f6f4 ClangReplInterpreterTests`clang::IncrementalExecutor::runCtors(this=0x0000aaaabadc2390) const at IncrementalExecutor.cpp:98:15
    frame #7: 0x0000aaaaadd26270 ClangReplInterpreterTests`clang::Interpreter::Execute(this=0x0000aaaabac90a60, T=0x0000aaaabac9ab90) at Interpreter.cpp:450:32
    frame #8: 0x0000aaaaadd26ea0 ClangReplInterpreterTests`clang::Interpreter::ParseAndExecute(this=0x0000aaaabac90a60, Code=(Data = "x", Length = 1), V=0x0000ffffffffeb70) at Interpreter.cpp:462:27
    frame #9: 0x0000aaaaac3ee9e0 ClangReplInterpreterTests`(anonymous namespace)::InterpreterTest_Value_Test::TestBody(this=0x0000aaaabac84930) at InterpreterTest.cpp:99:26
    frame #10: 0x0000aaaaad1d4d38 ClangReplInterpreterTests`void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=0x0000aaaabac84930, method=0x00000000000000010000000000000020, location="the test body") at gtest.cc:2612:10
    frame #11: 0x0000aaaaad1bcee8 ClangReplInterpreterTests`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x0000aaaabac84930, method=0x00000000000000010000000000000020, location="the test body") at gtest.cc:2667:12

And in __clang_Interpreter_SetValueNoAlloc, x3 is correct:

(lldb) register read x3
      x3 = 0x000000000000002a

And we see that it gets set before the call:

(lldb) dis
JIT(0xfffff7ff6000)`__stmts__0:
    0xfffff7ffa000 <+0>:  str    x30, [sp, #-0x10]!
    0xfffff7ffa004 <+4>:  adrp   x8, 0
    0xfffff7ffa008 <+8>:  mov    x0, #0xa60                ; =2656
    0xfffff7ffa00c <+12>: mov    x2, #0x72a0               ; =29344
    0xfffff7ffa010 <+16>: ldr    x8, [x8, #0x70]
    0xfffff7ffa014 <+20>: movk   x0, #0xbac9, lsl #16
    0xfffff7ffa018 <+24>: movk   x2, #0xbacc, lsl #16
    0xfffff7ffa01c <+28>: movk   x0, #0xaaaa, lsl #32
    0xfffff7ffa020 <+32>: movk   x2, #0xaaaa, lsl #32
    0xfffff7ffa024 <+36>: add    x1, x0, #0x40
    0xfffff7ffa028 <+40>: ldrsw  x3, [x8]
    0xfffff7ffa02c <+44>: bl     0xfffff7ffa078
->  0xfffff7ffa030 <+48>: ldr    x30, [sp], #0x10
    0xfffff7ffa034 <+52>: ret

Do we see anything like that in Arm's version?

│   0xe8bad000 <__stmts__0>                         push    {r11, lr}                                                                                                                         │
│   0xe8bad004 <__stmts__0+4>                       sub     sp, sp, #8                                                                                                                        │
│   0xe8bad008 <__stmts__0+8>                       ldr     r0, [pc, #44]   ; 0xe8bad03c <__stmts__0+60>                                                                                      │
│   0xe8bad00c <__stmts__0+12>                      mov     r2, #2672       ; 0xa70                                                                                                           │
│   0xe8bad010 <__stmts__0+16>                      orr     r2, r2, #847872 ; 0xcf000                                                                                                         │
│   0xe8bad014 <__stmts__0+20>                      ldr     r0, [pc, r0]                                                                                                                      │
│   0xe8bad018 <__stmts__0+24>                      ldr     r0, [r0]                                                                                                                          │
│   0xe8bad01c <__stmts__0+28>                      asr     r1, r0, #31                                                                                                                       │
│   0xe8bad020 <__stmts__0+32>                      strd    r0, [sp]                                                                                                                          │
│   0xe8bad024 <__stmts__0+36>                      ldr     r0, [pc, #12]   ; 0xe8bad038 <__stmts__0+56>                                                                                      │
│   0xe8bad028 <__stmts__0+40>                      orr     r1, r0, #32                                                                                                                       │
│   0xe8bad02c <__stmts__0+44>                      bl      0xe8bad060                                                                                                                        │
│  >0xe8bad030 <__stmts__0+48>                      add     sp, sp, #8                                                                                                                        │
│   0xe8bad034 <__stmts__0+52>                      pop     {r11, pc}                                                                                                                         │
│   0xe8bad038 <__stmts__0+56>                      muleq   r10, r8, r11                                                                                                                      │
│   0xe8bad03c <__stmts__0+60>                                      ; <UNDEFINED> instruction: 0xffffeffc

I need to look more to confirm but though there are loads they look to be close to the PC, and they don't write to r3. This leaves r3 set to the function address and causes the test failure.

[vgvassilev]

Adding @lhames and @weliveindetail for more wisdom about the JIT infrastructure.

[DavidSpickett]

FWIW, I am able to use printf fine on Arm, and that's variadic:

$ qemu-arm-static -L /work/open_source/arm-gnu-toolchain-13.2.Rel1-x86_64-arm-none-linux-gnueabihf/arm-none-linux-gnueabihf/libc/  ./bin/clang-repl
clang-repl> #include <stdio.h>
clang-repl> printf("%d\n", 42);
42
clang-repl> int x = 42;
clang-repl> x
Not implement yet.
clang-repl> printf("%d\n", 42);
42

I also thought maybe there is some circumstance where __clang_Interpreter_SetValueNoAlloc ends up called with no arguments. Perhaps if int x = 42; is actually processed as int x; then x = 42. The first call would be undefined behaviour as there's no argument to get from the varargs.

[vgvassilev]

Perhaps it is the whole typecasting business we do but you say that it is already wrong there.

In that case perhaps it is the way we take the ownership of the execution result around here

llvm-project/clang/lib/Interpreter/Interpreter.cpp

Line 709 in f11e08f

static ExprResult transformForValuePrinting(RuntimeInterfaceBuilder *Builder,

?

[vgvassilev]

Another place to look is here

llvm-project/clang/include/clang/Interpreter/Value.h

Line 143 in e7d569a

#define X(type, name) \

where we fetch from an union-like struct and maybe the casts went wrong but I do not see how…

[weliveindetail]

I cannot look into it more closely right now, but is there a difference between Arm and Thumb code? If so, it might be a relocation issue.

[DavidSpickett]

I tried adding -marm or -mthumb to the flags for building clang-repl, and the flags here:

llvm-project/clang/unittests/Interpreter/InterpreterTest.cpp

Line 45 in e5bdb7a

Args ClangArgs = {"-Xclang", "-emit-llvm-only"};

No combination of those fixed the issue. Assuming that's how I was supposed to try it. I did see __stmts__0's code change when Thumb was enabled in the unit test, so that did something at least.

[weliveindetail]

I did see __stmts__0's code change when Thumb was enabled in the unit test, so that did something at least

Yes, that's what I wanted to confirm. If it happens in both ISA modes, Arm and Thumb, then it's likely not an issue with the AArch32 backend in JITLink. They use different relocation types, indirection stubs, etc. and probably they aren't both broken the same way.

[weliveindetail]

If this issue is still open in 2 weeks, I can take a look :)