Compile Clang to lind-wasm

Open this issue to document how to compile clang to lind-wasm. The goal is to compile a Clang that has `--build=x86_64-linux`, `--host=wasm32`, `--target=x86_64-linux`, and linking to lind-glibc. 

# Compile libc++ library

## I. Scripts

### i. CMake

```cmake
# Toolchain-WASI.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR wasm32)

set(CLANG_BIN "/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin")
set(CMAKE_C_COMPILER "${CLANG_BIN}/clang")
set(CMAKE_CXX_COMPILER "${CLANG_BIN}/clang++")
set(CMAKE_LINKER "${CLANG_BIN}/bin/wasm-ld")
set(CMAKE_SYSROOT "/home/lind/lind-wasm/src/glibc/sysroot")

set(CMAKE_C_COMPILER_TARGET wasm32-unknown-wasi)
set(CMAKE_CXX_COMPILER_TARGET wasm32-unknown-wasi)

set(CMAKE_C_FLAGS_INIT "-static -nostdlib -nodefaultlibs -fno-exceptions -fno-unwind-tables")
set(CMAKE_CXX_FLAGS_INIT "-static -nostdlib -nodefaultlibs -stdlib=libc++ -fno-exceptions -fno-unwind-tables -fno-rtti")
set(CMAKE_EXE_LINKER_FLAGS_INIT "-static -nostdlib -nodefaultlibs")

# Optional: disable rpath injection
set(CMAKE_SKIP_RPATH ON)

# These fix platform error
set(LLVM_HOST_TRIPLE "wasm32-unknown-wasi")
set(LLVM_DEFAULT_TARGET_TRIPLE "x86_64-unknown-linux-gnu")

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
```

### ii. Build Script

```sh
#!/bin/bash
set -e

export ROOT_DIR=$(pwd)
export LLVM_SRC="$ROOT_DIR/llvm-project"
export INSTALL_PREFIX="$ROOT_DIR/libcxx-wasi-install"

mkdir -p libcxx-build
cmake -B libcxx-build -S "$LLVM_SRC/runtimes" \
  -DCMAKE_TOOLCHAIN_FILE="$ROOT_DIR/Toolchain-WASI.cmake" \
  -DLLVM_PATH="$LLVM_SRC/llvm" \
  -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
  -DLLVM_TARGETS_TO_BUILD="X86" \
  -DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-linux-gnu" \
  -DLLVM_HOST_TRIPLE="wasm32-unknown-wasi" \
  -DLIBCXX_ENABLE_SHARED=OFF \
  -DLIBCXX_ENABLE_STATIC=ON \
  -DLIBCXX_ENABLE_EXCEPTIONS=OFF \
  -DLIBCXX_USE_COMPILER_RT=ON \
  -DLIBCXXABI_ENABLE_SHARED=OFF \
  -DLIBCXXABI_ENABLE_STATIC=ON \
  -DLIBCXXABI_ENABLE_EXCEPTIONS=OFF \
  -DLIBCXXABI_USE_LLVM_UNWINDER=OFF \
  -DLIBCXXABI_ENABLE_STATIC_UNWINDER=OFF \
  -DLIBCXXABI_USE_COMPILER_RT=ON \
  -DLIBCXXABI_LIBCXX_PATH="$LLVM_SRC/libcxx" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX="$INSTALL_PREFIX" \
  -DCMAKE_CXX_COMPILER_WORKS=1 \
  -DCMAKE_C_COMPILER_WORKS=1

cmake --build libcxx-build --target install

nm -C "$INSTALL_PREFIX/lib/wasm32-wasi/libc++.a" | grep cxa_throw
```

### iii. Necessary modification before compilation

1. `/home/lind/lind-wasm/llvm-project/libcxx/src/filesystem/filesystem_common.h`

Change the instantiation site in `convert_to_timespec()` in line 456 to cast the arguments. 
This avoids the mismatch but assumes long and `__time64_t` are bit-compatible.

```cpp
return set_times_checked(reinterpret_cast<long*>(&dest.tv_sec),
                         reinterpret_cast<long*>(&dest.tv_nsec),
                         tp);
```

2. `/home/lind/lind-wasm/libcxx-build/include/c++/v1/__support/musl/xlocale.h`

Change line 40 to make function signature to `unsigned long long` to match the glibc version

_Notes: manually disable musl usage from compilation flags (`-D_LIBCPP_HAS_MUSL_LIBC` and `-D_LIBCPP_HAS_MUSL_LIBC`) doesn't work_

## II. Copy libraries to sysroot

```sh
cp -r /home/lind/lind-wasm/libcxx-wasi-install/include/c++ \
      /home/lind/lind-wasm/src/glibc/sysroot/include/wasm32-wasi/

cp /home/lind/lind-wasm/libcxx-wasi-install/lib/libc++.a \
   /home/lind/lind-wasm/libcxx-wasi-install/lib/libc++abi.a \
   /home/lind/lind-wasm/src/glibc/sysroot/lib/wasm32-wasi/
```

Necessary modifications after copying:

`/home/lind/lind-wasm/src/glibc/sysroot/include/wasm32-wasi/c++/v1/__support/musl/xlocale.h`:

Change line 40 to make function signature to `unsigned long long` to match the glibc version

_TODO: not sure why modifying the file on llvm source code doesn't influence_


# Compile `Clang-16` and `lld`

## I. `lind-glibc` modifications

### Align with POSIX standard

1. Add fields on `src/glibc/target/include/bits/struct_stat.h:53-58`: 

```c
struct stat
  {
    ...
    struct timespec st_atim;            /* Time of last access.  */
    struct timespec st_mtim;            /* Time of last modification.  */
    struct timespec st_ctim;            /* Time of last status change.  */
  };
```

2. Add one field of `/home/lind/lind-wasm/src/glibc/sysroot/include/wasm32-wasi/bits/statvfs.h:54`

```c
struct statvfs
  {
    ...
    unsigned int __f_type;
  };
```

### Add shim header file

#### `fix_std_maxmin.h`

Avoid the inconsistency of the two parameter types of `std::max` / `std::min`, which leads to the failure of template parameter _Tp deduction. The type deduction of libc++ adapted by wasi is more strict. Add `-include $FIX_HEADER` in CXX flags to force build using shim header file.

Add file at:

```sh
/home/lind/lind-wasm/src/glibc/sysroot/include/wasm32-wasi/c++/v1/__algorithm/fix_std_maxmin.h
```

```cpp
// fix_std_maxmin.h
#pragma once

#include <algorithm>
#include <type_traits>

namespace std {

template <class _Tp, class _Up>
constexpr typename std::common_type<_Tp, _Up>::type
max(const _Tp& __a, const _Up& __b) {
  return (__a < __b) ? __b : __a;
}

template <class _Tp, class _Up>
constexpr typename std::common_type<_Tp, _Up>::type
min(const _Tp& __a, const _Up& __b) {
  return (__b < __a) ? __b : __a;
}

}
```

### Add shim library

#### 0) Preparation and command line:

0.1) Create shim folder, and all shim library files are stored here

```sh
mkdir shim/
cd shim/
```

0.2) Compilation command line

All files are using same compilation command line (Remember to replace <shim_file> by actual file name!)

```sh
/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/clang \
    --target=wasm32-unknown-wasi -v -Wno-int-conversion -std=gnu11 -fgnu89-inline -matomics -mbulk-memory -O2 -g \
    -Wall -Wwrite-strings -Wundef -Wstrict-prototypes -Wold-style-definition \
    -fmerge-all-constants -ftrapping-math -fno-stack-protector -fno-common \
    -Wp,-U_FORTIFY_SOURCE -fmath-errno -fPIE -ftls-model=local-exec \
    -I/home/lind/lind-wasm/src/glibc/include \
    -I/home/lind/lind-wasm/src/glibc/build/nptl \
    -I/home/lind/lind-wasm/src/glibc/build \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/lind \
    -I/home/lind/lind-wasm/src/glibc/lind_syscall \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux/i386/i686 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux/i386 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux/x86/include \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux/x86 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/x86/nptl \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/i386/nptl \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux/include \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv/linux \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/nptl \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/pthread \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/gnu \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/inet \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/sysv \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix/i386 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/unix \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/posix \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/i386/fpu \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/x86/fpu \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/i386 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/x86/include \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/x86 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/wordsize-32 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754/float128 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754/ldbl-96/include \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754/ldbl-96 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754/dbl-64 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754/flt-32 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/ieee754 \
    -I/home/lind/lind-wasm/src/glibc/sysdeps/generic \
    -I/home/lind/lind-wasm/src/glibc \
    -I/home/lind/lind-wasm/src/glibc/libio \
    -I/home/lind/lind-wasm/src/glibc/math \
    -nostdinc -isystem /home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/lib/clang/16/include -isystem /usr/i686-linux-gnu/include \
    -D_LIBC_REENTRANT -include /home/lind/lind-wasm/src/glibc/build/libc-modules.h -DMODULE_NAME=libc \
    -include /home/lind/lind-wasm/src/glibc/include/libc-symbols.h -DPIC -DTOP_NAMESPACE=glibc \
    -o <shim_file>.o \
    -c <shim_file>.c
```

#### 1) `fenv_shim.a`: Shim library for `fetestexcept` / `feraiseexcept` / `fegetenv` / `fesetenv` / `fesetround`

_**Notes: For pure numerical calculations, the results of mathematical functions such as sqrt and pow will not be affected. However, the detection and handling of floating-point exception states will fail, so if we need to fine-tune the handling of calculation exceptions, a more complete implementation is required. Depending on the usage scenario of TriSeal (for example, this "missing exception" situation is usually tolerated in the WASM environment), a dummy implementation is usually feasible. If TriSeal does need to detect or respond to floating-point exceptions in the future, a more complex shim implementation may be required.**_

1.1) `fenv_shim.c`

```c
// fenv_shim.c
#include <fenv.h>

int fetestexcept(int e) { return 0; }
int feraiseexcept(int e) { return 0; }
int fegetenv(fenv_t *envp) { return 0; }
int fesetenv(const fenv_t *envp) { return 0; }
int fesetround(int r) { return 0; }
```

1.2) Use above compilation command to compile

1.3) Pack to library

```sh
# Pack to .a
/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-ar rcs libfenv_shim.a fenv_shim.o

# Put into sys root
cp libfenv_shim.a /home/lind/lind-wasm/src/glibc/sysroot/lib/wasm32-wasi/
```

#### 2) required math library

2.1) Compile from glibc source

```sh
cd src/glibc/sysdeps/ieee754/dbl-64/
```

2.2) Use above compilation command to compile e_exp2.c & e_fmod.c & math_err.c & e_exp_data.c

2.3) 

```sh
/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-ar rcs libm.a e_fmod.o e_exp2.o e_exp_data.o math_err.o

cp libm.a /home/lind/lind-wasm/src/glibc/sysroot/lib/wasm32-wasi/
```

#### 3) exception handling

Those are Definition of Exception Handling frame registration function, using dummy functions to replace since we've disabled libcunwind support.

3.1) 

```c
// eh_stub.c
void __register_frame(void* p) {}
void __deregister_frame(void* p) {}
```

3.2) Use above compilation command to compile 

3.3) 

```sh
/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-ar rcs libeh_stub.a eh_stub.o
# Put into same folder as libm.a
cp libeh_stub.a /home/lind/lind-wasm/src/glibc/sysroot/lib/wasm32-wasi/
```

#### 4) lll_elision

4.1)

```c
# lll_elision_shim.c
#include <stdint.h>

// fallback: always succeed
int __lll_lock_elision(int *futex, short *adapt_count, int trylock) {
    (void)futex;
    (void)adapt_count;
    (void)trylock;
    return 0;  // success
}

void __lll_unlock_elision(int *futex, short *adapt_count, int trylock) {
    (void)futex;
    (void)adapt_count;
    (void)trylock;
    // do nothing
}
```

4.2) Use above compilation command to compile 

4.3) 

```sh
/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-ar rcs lll_shim.a lll_elision_shim.o
cp lll_shim.a /home/lind/lind-wasm/src/glibc/sysroot/lib/wasm32-wasi/
```

## II. llvm source code

### Remove unused sub-project / sub-tools to prevent compilation error

```sh
rm -rf llvm-project/llvm/tools/llvm-isel-fuzzer llvm-project/llvm/tools/llvm-opt-fuzzer 
```

### Modify lld CMake file

This is a known issue that clang-lld will build all platforms. Modified CMake file of lld to only keep Common and ELF options.
`llvm-project/lld/CMakeLists.txt` and `llvm-project/lld/tools/lld/CMakeLists.txt`

```sh

## III. Scripts

### i. CMake

```cmake
# Toolchain-WASI.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR wasm32)

# Compiler and linker
set(CLANG_ROOT "/home/lind/lind-wasm/clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04")
set(CMAKE_C_COMPILER "${CLANG_ROOT}/bin/clang")
set(CMAKE_CXX_COMPILER "${CLANG_ROOT}/bin/clang++")
set(CMAKE_LINKER "${CLANG_ROOT}/bin/wasm-ld")

# Target configuration
set(CMAKE_C_COMPILER_TARGET wasm32-unknown-wasi)
set(CMAKE_CXX_COMPILER_TARGET wasm32-unknown-wasi)

# Sysroot for WASI environment
set(CMAKE_SYSROOT "/home/lind/lind-wasm/src/glibc/sysroot")

# Force CMake to accept compilers without try-run
set(CMAKE_C_COMPILER_WORKS TRUE)
set(CMAKE_CXX_COMPILER_WORKS TRUE)
set(CMAKE_EXECUTABLE_SUFFIX ".wasm")

# Don't pass -rpath to the linker
set(CMAKE_SKIP_RPATH TRUE)
set(CMAKE_SKIP_INSTALL_RPATH TRUE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH FALSE)

# Prevent try-run errors (because wasm can't run)
set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
```

### ii. Build Script

```sh
#!/bin/bash
set -e

export HOME_DIR="/home/lind/lind-wasm"
export LLVM_SRC="$HOME_DIR/llvm-project/llvm"
export BUILD_DIR="$HOME_DIR/llvm-wasm-build"
export TOOLCHAIN_FILE="$HOME_DIR/Toolchain-WASI-LLVM.cmake"
export LIBCXX_INCLUDE="$HOME_DIR/src/glibc/sysroot/include/wasm32-wasi/c++/v1"
export FIX_HEADER="/home/lind/lind-wasm/src/glibc/sysroot/include/wasm32-wasi/c++/v1/__algorithm/fix_std_maxmin.h"

mkdir -p "$BUILD_DIR"

cmake -B "$BUILD_DIR" -S "$LLVM_SRC" \
  -DCMAKE_TOOLCHAIN_FILE="$TOOLCHAIN_FILE" \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_PROJECTS="clang;lld" \
  -DLLVM_HOST_TRIPLE="wasm32-unknown-wasi" \
  -DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-linux-gnu" \
  -DLLVM_TARGETS_TO_BUILD="X86" \
  -DLLD_ENABLE_TARGETS="ELF" \
  -DCMAKE_C_FLAGS="-fno-exceptions -fno-unwind-tables" \
  -DCMAKE_CXX_FLAGS="-include $FIX_HEADER -fno-exceptions \
      -Wno-error=template-argument-type-deduction \
      -fno-unwind-tables -fno-rtti -I$LIBCXX_INCLUDE -D__GNU__ -D_POSIX_C_SOURCE=200809L" \
  -DCMAKE_EXE_LINKER_FLAGS="-nostdlib -nodefaultlibs \
      -L$HOME_DIR/src/glibc/sysroot/lib/wasm32-wasi \
      -Wl,--export=__stack_pointer,--export=__stack_low \
      -Wl,--import-memory,--export-memory \
      -Wl,--max-memory=67108864 \
      -lm -leh_stub -lfenv_shim" \
  -DLLVM_TOOL_LLI_BUILD=OFF \
  -DLLVM_TOOL_LLVM_JITLINK_EXECUTOR_BUILD=OFF \
  -DCMAKE_C_STANDARD_LIBRARIES="-lc -lcompiler_rt" \
  -DCMAKE_CXX_STANDARD_LIBRARIES="-lc++ -lc++abi -lcompiler_rt -lc" \
  -DCMAKE_INSTALL_RPATH="" \
  -DCMAKE_SKIP_RPATH=ON \
  -DCMAKE_SKIP_INSTALL_RPATH=ON \
  -DLLVM_ENABLE_THREADS=OFF \
  -DLLVM_ENABLE_PIC=OFF \
  -DLLVM_BUILD_SHARED_LIBS=OFF \
  -DLLVM_BUILD_TOOLS=OFF \
  -DLLVM_INCLUDE_TESTS=OFF \
  -DLLVM_INCLUDE_BENCHMARKS=OFF

cmake --build "$BUILD_DIR" 
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compile Clang to lind-wasm #4

Compile libc++ library

I. Scripts

i. CMake

ii. Build Script

iii. Necessary modification before compilation

II. Copy libraries to sysroot

Compile `Clang-16` and `lld`

I. `lind-glibc` modifications

Align with POSIX standard

Add shim header file

`fix_std_maxmin.h`

Add shim library

0) Preparation and command line:

1) `fenv_shim.a`: Shim library for `fetestexcept` / `feraiseexcept` / `fegetenv` / `fesetenv` / `fesetround`

2) required math library

3) exception handling

4) lll_elision

II. llvm source code

Remove unused sub-project / sub-tools to prevent compilation error

Modify lld CMake file

ii. Build Script

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Compile Clang to lind-wasm #4

Description

Compile libc++ library

I. Scripts

i. CMake

ii. Build Script

iii. Necessary modification before compilation

II. Copy libraries to sysroot

Compile Clang-16 and lld

I. lind-glibc modifications

Align with POSIX standard

Add shim header file

fix_std_maxmin.h

Add shim library

0) Preparation and command line:

1) fenv_shim.a: Shim library for fetestexcept / feraiseexcept / fegetenv / fesetenv / fesetround

2) required math library

3) exception handling

4) lll_elision

II. llvm source code

Remove unused sub-project / sub-tools to prevent compilation error

Modify lld CMake file

ii. Build Script

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Compile `Clang-16` and `lld`

I. `lind-glibc` modifications

`fix_std_maxmin.h`

1) `fenv_shim.a`: Shim library for `fetestexcept` / `feraiseexcept` / `fegetenv` / `fesetenv` / `fesetround`