Allow WASI-less builds with `--triple wasm32-unknown-unknown`

[MaxDesiatov]

Most probably depends on #2411, as ICU requires libc I guess? 🤔

Making this available as a separate triple allows us to ship both WASI-linked and WASI-less stdlib simultanously. Then people who need WASI (i.e. everyone who needs Foundation for example) can build with existing --triple wasm32-unknown-wasi, while --triple wasm32-unknown-unknown is available as an option for slim builds that don't need Foundation.

[kateinoigakukun]

Most probably depends on #2411, as ICU requires libc I guess? 🤔

Yes, ICU depends on wasi-libc and WASI API.

And anothet big blocker is that C++ part of stdlib heavily depends on libcxx and libc. So if we want to make a non-wasi binary, we need a wasi-less libc. If wasi-libc provide a subset of the libc APIs that are able to implement without WASI APIs, it would be a great first step.

[MaxDesiatov]

I guess the libcxx dependency is caused by Swift runtime? I wonder how easily that depedency could be removed, or should we just stick to libcxx and hope that DCE will make it slim enough?

[kateinoigakukun]

I think it's not easy to remove libcxx dependency completely. But libcxx has many options and it works on bare-metal environment, so I think libcxx can be built upon wasi-less libc.

[kateinoigakukun]

Here is linker errors when not passing -lc

wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umutex.ao): undefined symbol: __cxa_atexit
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/crt1.o: undefined symbol: __original_main
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/crt1.o: undefined symbol: __prepare_for_exit
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: abort
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(ReflectionMirror.cpp.o): undefined symbol: asprintf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(ucnv.ao): undefined symbol: atol
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(HeapObject.cpp.o): undefined symbol: calloc
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(LibcShims.cpp.o): undefined symbol: close
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: cos
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: cosf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Random.cpp.o): undefined symbol: errno
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: exp
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: exp2
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: exp2f
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: expf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: fclose
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: ferror
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: fma
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: fmaf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: fmod
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: fmodf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: fopen
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: fprintf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: fread
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: free
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: fseek
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(umapfile.ao): undefined symbol: ftell
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: fwrite
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Random.cpp.o): undefined symbol: getentropy
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(EnvironmentVariables.cpp.o): undefined symbol: getenv
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: getline
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: gettimeofday
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: gmtime
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(uloc.ao): undefined symbol: isspace
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: localtime
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: log
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: log10
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: log10f
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: log2
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: log2f
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: logf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: malloc
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: malloc_usable_size
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Metadata.cpp.o): undefined symbol: memchr
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Metadata.cpp.o): undefined symbol: memcmp
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(KnownMetadata.cpp.o): undefined symbol: memcpy
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: memmove
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(KnownMetadata.cpp.o): undefined symbol: memset
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: mktime
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: modf
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(thread.cpp.obj): undefined symbol: nanosleep
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: newlocale
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: nl_langinfo
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Random.cpp.o): undefined symbol: open
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Heap.cpp.o): undefined symbol: posix_memalign
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: pow
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: putc
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(LibcShims.cpp.o): undefined symbol: putchar_unlocked
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(ucurr.ao): undefined symbol: qsort
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(LibcShims.cpp.o): undefined symbol: read
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(SmallVector.cpp.o): undefined symbol: realloc
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: remainder
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: remainderf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: round
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: roundf
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(memory.cpp.obj): undefined symbol: sched_yield
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: setlocale
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: sin
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Swift.o): undefined symbol: sinf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: snprintf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(uloc.ao): undefined symbol: sprintf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: stderr
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: stdin
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(LibcShims.cpp.o): undefined symbol: stdout
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: strcasecmp
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: strcat
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(DynamicCast.cpp.o): undefined symbol: strchr
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Metadata.cpp.o): undefined symbol: strcmp
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Metadata.cpp.o): undefined symbol: strcpy
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: strdup
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(system_error.cpp.obj): undefined symbol: strerror_r
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Demangle.cpp.o): undefined symbol: strlen
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: strncat
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(DynamicCast.cpp.o): undefined symbol: strncmp
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: strncpy
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(ReflectionMirror.cpp.o): undefined symbol: strndup
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: strrchr
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(ucnvmbcs.ao): undefined symbol: strstr
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(uloc.ao): undefined symbol: strtod
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: strtod_l
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: strtof
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: strtof_l
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(EnvironmentVariables.cpp.o): undefined symbol: strtol
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: strtold
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: strtold_l
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: strtoll
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(cstring.ao): undefined symbol: strtoul
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: strtoull
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: swprintf
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(thread.cpp.obj): undefined symbol: sysconf
wasm-ld: error: /usr/lib/swift_static/wasi/libicuuc.a(putil.ao): undefined symbol: time
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Stubs.cpp.o): undefined symbol: uselocale
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: vasprintf
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++abi.a(abort_message.cpp.o): undefined symbol: vfprintf
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(ErrorHandling.cpp.o): undefined symbol: vsnprintf
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcslen
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstod
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstof
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstol
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstold
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstoll
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstoul
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wcstoull
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wmemchr
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wmemcmp
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wmemcpy
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wmemmove
wasm-ld: error: /usr/share/wasi-sysroot/lib/wasm32-wasi/libc++.a(string.cpp.obj): undefined symbol: wmemset
wasm-ld: error: /usr/lib/swift_static/wasi/libswiftCore.a(Errors.cpp.o): undefined symbol: write

And here is imports list after DCEed by wasm-opt

Import[15]:
 - func[0] sig=13 <wasi_snapshot_preview1.proc_exit> <- wasi_snapshot_preview1.proc_exit
 - func[1] sig=10 <wasi_snapshot_preview1.args_sizes_get> <- wasi_snapshot_preview1.args_sizes_get
 - func[2] sig=10 <wasi_snapshot_preview1.args_get> <- wasi_snapshot_preview1.args_get
 - func[3] sig=4 <wasi_snapshot_preview1.fd_read> <- wasi_snapshot_preview1.fd_read
 - func[4] sig=4 <wasi_snapshot_preview1.fd_write> <- wasi_snapshot_preview1.fd_write
 - func[5] sig=11 <wasi_snapshot_preview1.fd_close> <- wasi_snapshot_preview1.fd_close
 - func[6] sig=10 <wasi_snapshot_preview1.fd_fdstat_get> <- wasi_snapshot_preview1.fd_fdstat_get
 - func[7] sig=128 <wasi_snapshot_preview1.path_open> <- wasi_snapshot_preview1.path_open
 - func[8] sig=10 <wasi_snapshot_preview1.fd_prestat_get> <- wasi_snapshot_preview1.fd_prestat_get
 - func[9] sig=1 <wasi_snapshot_preview1.fd_prestat_dir_name> <- wasi_snapshot_preview1.fd_prestat_dir_name
 - func[10] sig=10 <wasi_snapshot_preview1.random_get> <- wasi_snapshot_preview1.random_get
 - func[11] sig=10 <wasi_snapshot_preview1.environ_sizes_get> <- wasi_snapshot_preview1.environ_sizes_get
 - func[12] sig=10 <wasi_snapshot_preview1.environ_get> <- wasi_snapshot_preview1.environ_get
 - func[13] sig=10 <wasi_snapshot_preview1.fd_fdstat_set_flags> <- wasi_snapshot_preview1.fd_fdstat_set_flags
 - func[14] sig=132 <wasi_snapshot_preview1.fd_seek> <- wasi_snapshot_preview1.fd_seek

This shows we only uses WASI API in CommandLine.swift, OutputStream.swift, Random.swift and EnvironmentVariables.cpp. So if we can build libc without WASI, it's really easy to remove WASI dependency.

[MaxDesiatov]

What do you think of metallic as a WASI-libc replacement? I also had a look at rusl, but that seems abandoned and not focused on WebAssembly at all, although I would prefer a Rust implementation if one existed.

[MaxDesiatov]

Our maybe if we prefer a libc written C it would make more sense to fork WASI-libc and strip it of WASI dependencies and stuff we don't need in WASI-less builds, like time/threads/FS etc?

[kateinoigakukun]

@MaxDesiatov Thanks for researching 😄 I took a look at metallic, but it's designed to be used with a JavaScript environment, so it doesn't match our requirements...

I wondered some ways to provide wasi-less libc:

1. Start a new project to implement a libc for wasm32-unknown-unknown target
2. Fork and remove wasi related code from wasi-libc ( and send PRs to upstream wasi-libc repo)
3. Don't fork wasi-libc and maintain just a build script repo.

[MaxDesiatov]

With the option 3 would we just avoid using WASI-related APIs and rely on DCE to exclude them? Or would a build script exclude them somehow? I think a combination of 2 and 3 looks most reliable long term.

[kateinoigakukun]

Fortunately, musl and cloudlibc implements each libc API in separated files, so the build script can exclude WASI-related APIs easily and don't have to rely on DCE, I think. So with option 3, only what we need to do is to write a Makefile.

[MaxDesiatov]

BTW, Saleem previously mentioned he'd like to see WASI-libc to be compatible with CMake for it be properly used in upstream Swift, so maybe what we really need to write is a CMakeLists.txt file 😄

[filip-sakel]

Could the removal of the ICU dependency allow for WASI-less builds?

[MaxDesiatov]

I think, potentially yes. There's already some FREESTANDING configuration in the toolchain, but I don't know how stable it is. Support for the WASI-less triple would most probably be based on that configuration.

[filip-sakel]

Is there some way I could use this triple with carton dev?

[MaxDesiatov]

Unfortunately, not until this is supported in the toolchain itself