Zephyr build fail with LLVM on Windows

[yanzhouy]

Describe the bug
I am trying to build zephyr project in Windows with llvm compiler, but it failed.

To Reproduce
Steps to reproduce the behavior:
I installed llvm version 11.0.0 for windows and set the patch to the location.
In zephyr/samples/hello_world directory, delete all the /build content,

1. set ZEPHYR_TOOLCHAIN_VARIANT=llvm
2. set LLVM_TOOLCHAIN_PATH=C:\tools\windows\toolchains\llvm_11_0_0\
3. west build -b qemu_x86 .
4. See error: Re-running CMake. It seems to go to infinite loop and cannot build the application.

Expected behavior
Build can be done for hello-world as cross-compile with gcc i486.

Logs and console output

-- The C compiler identification is Clang 11.0.0
-- The CXX compiler identification is Clang 11.0.0
-- The ASM compiler identification is Clang

-- Configuring done
-- Generating done
-- Build files have been written to: /zephyr/zephyr-intel/samples/hello_world/build
[0/1] Re-running CMake...
Including boilerplate (Zephyr base (cached)):
...
[0/1] Re-running CMake...
...
[0/1] Re-running CMake...

Environment (please complete the following information):

• OS: Windows
• Toolchain: llvm

Additional context
Tried LLVM and Intel oneApi ICX compiler in Linux, both of them work.
Also tried with 'set ZEPHYR_TOOLCHAIN_VARIANT=cross-compile' and 'set CROSS_COMPILE=path to \ish_gcc_8_3\bin\i486-elf-' on Windows, samples/hello_world can be built successfully by 'west build -b qemu_x86 . '
How to build Zephyr project with clang or icx compiler on Windows?

[nashif]

Windows support is planned and is not done yet.

[yanzhouy]

Windows support is planned and is not done yet.

@nashif Hi Nashif, many thanks for your reply. When will zephyr could support llvm on windows platform? Do we have an ETA? Or maybe is there any workaround which I can do for using llvm compiler for zephyr on windows?

[nashif]

@tejlmand any idea what is going on here and why we go into a loop?

[yanzhouy]

**Also tried with latest LLVM toolchain, error reported below:

CMake Error at ../../cmake/extensions.cmake:1735 (message):
Assertion failed: The toolchain is unable to build a dummy C file. See
CMakeError.log.
Call Stack (most recent call first):
../../CMakeLists.txt:42 (assert)

**In CMakeError.log, it shows that:

Checking whether the ASM compiler is GNU using "--version" did not match "(GNU assembler)|(GCC)|(Free Software Foundation)":
clang version 11.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:/Users/yanzhouy/source/ish/ish_pic_base/ish/firmware/tools/windows/toolchains/llvm_11_0_0/bin

[yanzhouy]

hi @tejlmand Attached more log for the build error, could you please give some help for llvm build on Windows？ Many thanks.

[nashif]

i finally got to it and I am able to reproduce the recursive issue.

[yanzhouy]

Hi @nashif
Mostly, when I tried with 'set ZEPHYR_TOOLCHAIN_VARIANT=llvm' and 'set LLVM_TOOLCHAIN_PATH=C:\tools\windows\toolchains\llvm_11', Then I built with 'west build -b qemu_x86 .' for samples/hello_world.

It got the below CMake error reported:
CMake Error at ../../cmake/extensions.cmake:1735 (message): Assertion failed: The toolchain is unable to build a dummy C file. See CMakeError.log.

In ../../cmake/extensions.cmake:1735, it has the code as:

And in CMakeError.log, it shows:
Checking whether the ASM compiler is GNU using "--version" did not match "(GNU assembler)|(GCC)|(Free Software Foundation)":
clang version 11.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:/Users/yanzhouy/source/ish/pic_base/ish/firmware/tools/windows/toolchains/llvm_11/bin

[nashif]

after fixing the path of system includes in #33945 the errors are about missing gcc. Looks like on windows llvm depends on gcc or an msvc environment.

[yanzhouy]

Yes, it seems that Clang attempts to use MSVC's standard library on Windows by default, or GCC's libraries, since Clang's own standard library doesn't work on Windows yet.

[tejlmand]

For some reason i'm not able to re-produce the infinite loop when using llvm, but I oberserve the infinite CMake loop when working on arm-clang support in windows, and I suspect those to be related.

But what I do observe during build when using clang on windows is this:

In file included from C:/projects/github/ncs/zephyr/arch/posix/core/offsets/offsets.c:26:
In file included from ../include\kernel.h:17:
In file included from ../include\kernel_includes.h:19:
../include\toolchain.h:48:10: fatal error: 'toolchain/other.h' file not found
#include <toolchain/other.h>
         ^~~~~~~~~~~~~~~~~~~
1 error generated.
ninja: build stopped: subcommand failed.

zephyr/include/toolchain.h

Lines 40 to 47 in 5cc20fa

	#elif defined(__GNUC__) || (defined(_LINKER) && defined(__GCC_LINKER_CMD__))
	#include <toolchain/gcc.h>
	#else
	/* This include line exists for off-tree definitions of compilers,
	* and therefore this header is not meant to exist in-tree
	*/
	#include <toolchain/other.h>
	#endif

On Linux, clang identifies itself as __GNUC__ compatible.
Run: clang -dM -E -x c /dev/null and you'll see (among others):

#define __GNUC__ 4
#define __clang__ 1
#define __llvm__ 1

but on windows, clang reports, clang.exe -dM -E -x c dummy.c:

#define _MSC_BUILD 1
#define _MSC_VER 1913
#define __clang__ 1
#define __llvm__ 1

and thus not __GNUC__ compatible.

And a quick search in Zephyr repo reveals 60 occurrences of __GNUC__:

$  grep -rw __GNUC__|wc -l
60

[yanzhouy]

For current clang on Windows, use clang -v, shows:
Target: x86_64-pc-windows-msvc.
But for the build, the target should be x86_64-w64-windows-gnu.

[tejlmand]

For current clang on Windows, use clang -v, shows:
Target: x86_64-pc-windows-msvc.
But for the build, the target should be x86_64-w64-windows-gnu.

Thanks for the hint.
Of course we must setup the triplet correctly on windows, which apparently is not correctly done.

When doing clang -target i686-pc-none-elf -dM -E -x c dummy.c I get the desired behavior.
i686-pc-none-elf is the triplet in use for qemu_x86 in Linux.

#define __GNUC__ 4
#define __clang__ 1
#define __llvm__ 1

but for unknown reason this triplet is not applied when building on windows.
or x86_64 depending on target.

[tejlmand]

but for unknown reason this triplet is not applied when building on windows.

ok, seems to be correctly applied in later CMake versions.
Fails with CMake 3.17, but correctly applied when using CMake 3.20 on windows.

[tejlmand]

Seems we need to look at what to do for the gcc assembly:

.... zephyr/arch/x86/core/ia32/crt0.S
C:/projects/github/ncs/zephyr/arch/x86/core/ia32/crt0.S: Assembler messages:
C:/projects/github/ncs/zephyr/arch/x86/core/ia32/crt0.S:28: Warning: .type pseudo-op used outside of .def/.endef: ignored.
C:/projects/github/ncs/zephyr/arch/x86/core/ia32/crt0.S:28: Error: junk at end of line, first unrecognized character is `_'
....
clang: error: assembler command failed with exit code 1 (use -v to see invocation)

which on my system happens because clang can't use it's built-in assembler and thus calls out to an external assembler, in my particular case: C:\\cygwin64\\bin\\as.exe

 "C:\\cygwin64\\bin\\as.exe" --32 -I ../kernel/include -I ../arch/x86/include -I ../include -I zephyr/include/generated -I ../soc/x86/ia32 -o zephyr/arch/arch/x86/core/CMakeFiles/arch__x86__core.dir/ia32/crt0.S.obj "...\\AppData\\Local\\Temp\\crt0-b2d06c.s"

and trying to avoid that results in:

clang: error: there is no external assembler that can be used on this platform

[tejlmand]

which on my system happens because clang can't use it's built-in assembler and thus calls out to an external assembler, in my particular case: C:\\cygwin64\\bin\\as.exe

Seems to be caused by this:

zephyr/arch/x86/core/ia32.cmake

Lines 4 to 6 in 35f52fd

	if (CMAKE_C_COMPILER_ID STREQUAL "Clang")
	# We rely on GAS for assembling, so don't use the integrated assembler
	zephyr_compile_options($<$<COMPILE_LANGUAGE:ASM>:-no-integrated-as>)

[yanzhouy]

hi @tejlmand

I reproduced the error on qemu_x86 board in zephyr like below:

But I installed the old version of clang for my windows 10: https://releases.llvm.org/3.7.0/LLVM-3.7.0-win64.exe
It has the 'Target: x86_64-w64-windows-gnu'.
Also, tried to build with qemu_x86_64 as the target board, did not show the above error like qemu_x86 but it has link error:

[yanzhouy]

Tried with following process:

Modified in zephyr/cmake/compiler/clang/target.cmake (line 60):

execute_process(
COMMAND i486-elf-gcc --print-libgcc-file-name
OUTPUT_VARIABLE LIBGCC_FILE_NAME
OUTPUT_STRIP_TRAILING_WHITESPACE
)

And added triple in zephyr/cmake/toolchain/llvm/generic.cmake (line 25):

set(triple i486-elf)
set(CMAKE_C_COMPILER_TARGET ${triple})
set(CMAKE_ASM_COMPILER_TARGET ${triple})
set(CMAKE_CXX_COMPILER_TARGET ${triple})

Add gcc_8_3\bin which contains 'i486-elf-' in the system environment path, build for target qemu_x86 with llvm on windows can pass.

CMake version: 3.20.1
Clang version: 3.7.1

set ZEPHYR_TOOLCHAIN_VARIANT=llvm
set LLVM_TOOLCHAIN_PATH=path to toolchains\LLVM_3_7_1
west build -b qemu_x86 .

[carlescufi]

@YanzhouYu can you please retry with the current master?

[yanzhouy]

Hi @carlescufi @tejlmand @nashif

I tried with latest zephyr code v2.6.0-rc1 on Windows 10,
Run command in cmd with administrator access:

set ZEPHYR_TOOLCHAIN_VARIANT=llvm
set LLVM_TOOLCHAIN_PATH=path to toolchains\LLVM_12
west build -b qemu_x86 samples/hello_world 

CMake reports error:
Assertion failed: The toolchain is unable to build a dummy C file.

After comment 'assert' in CMakeLists.txt line 42:

# Verify that the toolchain can compile a dummy file, if it is not we
# won't be able to test for compatibility with certain C flags.
zephyr_check_compiler_flag(C "" toolchain_is_ok)
# assert(toolchain_is_ok "The toolchain is unable to build a dummy C file. See CMakeError.log.")

Build goes to infinite loop with Re-running CMake:

Could you please take a look at this problem?
Which version should I use for enabling LLVM on Windows? (Currently I test with latest LLVM version 12.0.0)

[chen-png]

CMake reports error:
Assertion failed: The toolchain is unable to build a dummy C file.

After comment 'assert' in CMakeLists.txt line 42:

Build goes to infinite loop with Re-running CMake

got the same issue on windows with the latest main branch aec4c49

[chen-png]

waiting for the "infinite loop with Re-running CMake" and it will report an error as below finally:
ninja: error: manifest 'build.ninja' still dirty after 100 tries

then run the command "ninja -C build -d explain" to get details, it reported
ninja explain: output build.ninja older than most recent input C: (6465940461028307 vs 6465940461816204)

it seems that ninja will always judge the build.ninja is not the latest, so it always regenerated it.

[chen-png]

hi @tejlmand Could you help take a look of this problem?
I installed a mingw for clang on windows, and ran the command "west build -b qemu_x86 -p auto samples\hello_world -c" to build tests, it failed with "mingw32/bin/ld.exe: cannot perform PE operations on non PE output file 'zephyr\zephyr_prebuilt.elf'.
I found the clang will use the ld of mingw in default, rather than lld from LLVM, how can I change the default linker path?

FAILED: zephyr/zephyr_prebuilt.elf zephyr/zephyr_prebuilt.map cmd.exe /C "cd . && C:\Users\chenp1\Documents\LLVM_3.7\LLVM\bin\clang.exe --target=i686-pc-none-elf zephyr/CMakeFiles/zephyr_prebuilt.dir/misc/empty_file.c.obj -o zephyr\zephyr_prebuilt.elf -Wl,-T zephyr/linker_zephyr_prebuilt.cmd -Wl,-Map=C:/Users/chenp1/Documents/zephyrproject/zephyr/build/zephyr/zephyr_prebuilt.map -Wl,--whole-archive app/libapp.a zephyr/libzephyr.a zephyr/arch/arch/x86/core/libarch__x86__core.a zephyr/lib/libc/minimal/liblib__libc__minimal.a zephyr/lib/posix/liblib__posix.a zephyr/subsys/random/libsubsys__random.a zephyr/drivers/serial/libdrivers__serial.a -Wl,--no-whole-archive zephyr/kernel/libkernel.a zephyr/CMakeFiles/offsets.dir/./arch/x86/core/offsets/offsets.c.obj -L"C:/Users/chenp1/mingw/lib/gcc/mingw32/6.3.0" -LC:/Users/chenp1/Documents/zephyrproject/zephyr/build/zephyr -lgcc -Wl,--gc-sections -Wl,--build-id=none -Wl,--sort-common=descending -Wl,--sort-section=alignment -Wl,-u,_OffsetAbsSyms -Wl,-u,_ConfigAbsSyms -nostdlib -static -Wl,-X -Wl,-N -Wl,--orphan-handling=warn && cmd.exe /C "cd /D C:\Users\chenp1\Documents\zephyrproject\zephyr\build\zephyr && C:\Users\chenp1\Documents\CMake\bin\cmake.exe -E echo "" c:/users/chenp1/mingw/bin/../lib/gcc/mingw32/6.3.0/../../../../mingw32/bin/ld.exe: cannot perform PE operations on non PE output file 'zephyr\zephyr_prebuilt.elf'.

[chen-png]

the compiler clang of LLVM will used external system linker in default, i.e, the linker of mingw or VC, not the ld.lld of LLVM built-in, so in my environment, it used the mingw32/bin/ld.exe, this is not a cross-compiler and I found it only support i386pe, couldn't be used to build zephyr elf file.
so tried to add -fuse-ld=lld option to force clang to use the LLVM built-in linker, but it reported some new errors as below:

ld.lld: error: zephyr/linker_zephyr_prebuilt.cmd:29: malformed number: "__start"
epoint = (("__start") - ((0x0 + 0x100000) - (0x0 + 0x100000)));

and I tested to use LLVM built-in linker on linux, it reported the same error.

[chen-png]

CMake reports error:
Assertion failed: The toolchain is unable to build a dummy C file.

this error was caused that cmake couldn't find a valid linker path when checking compiler support flag, because clang won't use LLVM built-in lld in default.

Build goes to infinite loop with Re-running CMake

the infinite loop, please use the ninja version 1.8.2, not the latest.

[chen-png]

@YanzhouYu I tested the LLVM built-in lld on windows and linux, it reported the same error, it seems that it couldn't support cross linker well ,so I replaced it by a cross i486-elf.ld.exe to complete link process, it's ok to generate final zpehyr.elf, so please use a cross gcc linker for link process firstly.

[chen-png]

tried to install Intel OneApi toolchain on Windows, but I found that icx will invoke clang-cl, rather than the clang on Windows, and the clang-cl is used to be compatible with msvc cl compiler, there are some different options with clang, so when I used icx to build zephyr binaries on windows, it reported some errors like below, because clang-cl doesn't have those options "-x assembler-with-cpp","-isystem","-include".

clang-cl: warning: unknown argument ignored in clang-cl: '-x' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-isystem' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-include' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-MF' [-Wunknown-argument]
clang-cl: error: no such file or directory: 'assembler-with-cpp'

[chen-png]

I tried to change to use clang from oneApi directly on Windows by setting compiler name as clang, not icx.

set(COMPILER clang)

but it met a new error, I found that the cross compiling target set by CMAKE_C_COMPILER_TARGET has not been passed to clang,
i.e for building a test on qemu_x86 on windows, clang command should be like this “clang --target=i686-pc-none-elf” , however, when I used the clang from OneAPI, this option won’t be added after clang command, it used the default target “x86_64-pc-windows-msvc”.

the correct command should be like this:

[chen-png]

also tried to use other cmake functions "add_compiler_options, add_link_options" to solve the CMAKE_C_COMPILER_TARGET recognition problem above.

add_compile_options(--target=${triple})
add_link_options(--target=${triple})

It could continue to build and compile those C source files, but I don't know why it skipped those assembly files.
i.e when building a zephyr library lib__arch__x86__core.a, according to the cmake script below, it should compile these 6 files “crt0.S, excstub.S, intstub.S, irq_manage.c, swap.S, thread.c", but I found that it only compiled C source files.

the original cmake script as below:

if (CMAKE_C_COMPILER_ID STREQUAL "Clang")
zephyr_compile_options($<$<COMPILE_LANGUAGE:ASM>:-no-integrated-as>)
elseif(CMAKE_C_COMPILER_ID STREQUAL "GNU")
zephyr_compile_options($<$<COMPILE_LANGUAGE:ASM>:-Wa,--divide>)
endif()
zephyr_library_sources(
ia32/crt0.S
ia32/excstub.S
ia32/intstub.S
ia32/irq_manage.c
ia32/swap.S
ia32/thread.c
)

[chen-png]

fixed by #37157.