LLVM built application crash

[nashif]

Describe the bug
hello_world built with llvm crashes.

To Reproduce

• export ZEPHYR_TOOLCHAIN_VARIANT=llvm
• west build -b qemu_x86 samples/hello_world/ -t run

[117/118] To exit from QEMU enter: 'CTRL+a, x'[QEMU] CPU: qemu32,+nx,+pae
SeaBIOS (version rel-1.12.1-0-ga5cab58-dirty-20200625_115407-9426dddc0a1f-zephyr
)
Booting from ROM..FAILED: zephyr/CMakeFiles/run
cd /home/nashif/Work/zephyrproject/zephyr/build && /home/nashif/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/bin/qemu-system-i386 -m 9 -cpu qemu32,+nx,+pae -device isa-debug-exit,iobase=0xf4,iosize=0x04 -no-reboot -nographic -net none -pidfile qemu.pid -chardev stdio,id=con,mux=on -serial chardev:con -mon chardev=con,mode=readline -icount shift=5,align=off,sleep=off -rtc clock=vm -kernel /home/nashif/Work/zephyrproject/zephyr/build/zephyr/zephyr.elf
ninja: build stopped: subcommand failed.

Expected behavior
it should run

Impact
All llvm built applications are crashing.

Environment (please complete the following information):

• OS: Linux
• Toolchain host clang/llvm

i9:zephyr(master): clang --version
clang version 10.0.1 (Fedora 10.0.1-2.fc32)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

• Commit 82dbda3

Additional context
Bisected tree and found this commit to be the cause:

commit bb99422c8a5f97d250e3e7c956c57ec78fae0e77
Author: Peter Bigot <peter.bigot@nordicsemi.no>
Date:   Sun Nov 8 19:57:52 2020 -0600

    lib/os: replace z_vprintk with cbprintf

    Using the same implementation as the rest of Zephyr reduces code size.

    Update options and expected results for formatting test.

    Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>

[pabigot]

This appears to be an LLVM code generation bug.

The problem goes away with this patch:

diff --git a/lib/os/cbprintf_complete.c b/lib/os/cbprintf_complete.c
index df3692e5d8..ee6bbb8df2 100644
--- a/lib/os/cbprintf_complete.c
+++ b/lib/os/cbprintf_complete.c
@@ -1351,6 +1351,10 @@ static int outs(cbprintf_cb out,
        return (int)count;
 }
 
+static union argument_value value = (union argument_value){
+       .uint = 0,
+};
+
 int cbvprintf(cbprintf_cb out, void *ctx, const char *fp, va_list ap)
 {
        char buf[CONVERTED_BUFLEN];
@@ -1456,7 +1460,7 @@ int cbvprintf(cbprintf_cb out, void *ctx, const char *fp, va_list ap)
                        = (enum specifier_cat_enum)conv.specifier_cat;
                enum length_mod_enum length_mod
                        = (enum length_mod_enum)conv.length_mod;
-               union argument_value value = (union argument_value){
+               value = (union argument_value){
                        .uint = 0,
                };

The observable impact of this is that without it llvm generates the following object:

 .rodata..L__const.cbvprintf.value
                0x0000000000104174        0xc zephyr/libzephyr.a(cbprintf_complete.c.obj)

rather than placing the union that holds the values read from the arguments onto the stack. Presumably it mostly replaces the va_arg() material with pointers to what's on the stack, but fails to substitute all of them leaving some reference that pulls data out of the const zero object, resulting in a fault.

Do we want to move the stack variable into RAM to accommodate LLVM? I'd be concerned the underlying problem would show up somewhere else.... of course, that wouldn't work as multiple format operations can be active simultaneously.

tirzah[170]$ clang --version
clang version 11.0.0 (git@github.com:llvm/llvm-project.git 279922f108c26fe09667ba2525ab18b4735b28b2)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin

[nashif]

@pabigot thanks for looking into this. We need a workaround for this until llvm is fixed, if this turns to be the culprit. We should file a bug against llvm for this.