Different libc.a for macOS arm64 and Linux x86_64

[eagleoflqj]

I'm testing reproducibility by compiling on Linux and macOS fcitx-contrib/fcitx5-js#18. Everything works fine except for .wasm. A wasm2wat and diff produces below and more

317,326c317,326
<   (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;139;) (type 1)))
<   (import "wasi_snapshot_preview1" "environ_get" (func (;140;) (type 1)))
<   (import "env" "__syscall_fcntl64" (func (;141;) (type 4)))
<   (import "env" "__syscall_ioctl" (func (;142;) (type 4)))
<   (import "env" "__syscall_openat" (func (;143;) (type 6)))
<   (import "wasi_snapshot_preview1" "fd_close" (func (;144;) (type 0)))
<   (import "wasi_snapshot_preview1" "fd_read" (func (;145;) (type 6)))
<   (import "wasi_snapshot_preview1" "fd_write" (func (;146;) (type 6)))
<   (import "env" "_tzset_js" (func (;147;) (type 8)))
<   (import "wasi_snapshot_preview1" "proc_exit" (func (;148;) (type 3)))
---
>   (import "wasi_snapshot_preview1" "proc_exit" (func (;139;) (type 3)))
>   (import "wasi_snapshot_preview1" "environ_sizes_get" (func (;140;) (type 1)))
>   (import "wasi_snapshot_preview1" "environ_get" (func (;141;) (type 1)))
>   (import "env" "__syscall_fcntl64" (func (;142;) (type 4)))
>   (import "env" "__syscall_ioctl" (func (;143;) (type 4)))
>   (import "env" "__syscall_openat" (func (;144;) (type 6)))
>   (import "wasi_snapshot_preview1" "fd_close" (func (;145;) (type 0)))
>   (import "wasi_snapshot_preview1" "fd_read" (func (;146;) (type 6)))
>   (import "wasi_snapshot_preview1" "fd_write" (func (;147;) (type 6)))
>   (import "env" "_tzset_js" (func (;148;) (type 8)))
923c923
<     call 2185
---
>     call 2186

I also tested the link command below with EMCC_DEBUG. Order of imports differs from /tmp/emscripten_temp/emcc-00-base.wasm.

It seems imports are sorted in test as well.

emscripten/test/test_other.py

Line 8864 in 5ac384b

imports.sort()

Not sure if it's a emscripten issue or llvm issue.

Version of emscripten/emsdk:
Please include the output emcc -v here

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.74 (1092ec30a3fb1d46b1782ff1b4db5094d3d06ae5)
clang version 20.0.0git (https:/github.com/llvm/llvm-project 322eb1a92e6d4266184060346616fa0dbe39e731)
Target: wasm32-unknown-emscripten
Thread model: posix

Failing command line in full:
Not failing.

Full link command and output with -v appended:

$ em++ -v -O3 -DNDEBUG  src/CMakeFiles/Fcitx5.dir/fcitx.cpp.o src/CMakeFiles/Fcitx5.dir/keycode.cpp.o src/CMakeFiles/
Fcitx5.dir/input_method.cpp.o src/CMakeFiles/Fcitx5.dir/config.cpp.o src/CMakeFiles/Fcitx5.dir/action.cpp.o src/CMakeFiles/Fcitx5.dir/event_js.cpp.o src/CMakeFiles/Fcitx
5.dir/__/dummy.cpp.o src/CMakeFiles/Fcitx5.dir/Fcitx5-webpanel-import-addon.cpp.o src/CMakeFiles/Fcitx5.dir/Fcitx5-wasmfrontend-import-addon.cpp.o src/CMakeFiles/Fcitx5.
dir/Fcitx5-keyboard-import-addon.cpp.o -o src/Fcitx5.js  -sMAIN_MODULE=1  -fexceptions  -sSTACK_SIZE=1MB  -sALLOW_MEMORY_GROWTH=1  -sMAXIMUM_MEMORY=4GB  -sEXPORTED_RUNTI
ME_METHODS=['ccall','FS']  -sENVIRONMENT=web  --extern-pre-js ../build/pre.js  --extern-post-js ../build/index.js  -L../build/destdir/usr/lib  --preload-file ../build/sy
sroot/usr/share/X11@/usr/share/X11  --preload-file ../build/sysroot/usr/share/iso-codes/json/iso_3166-1.json@/usr/share/iso-codes/json/iso_3166-1.json  --preload-file ..
/build/sysroot/usr/share/iso-codes/json/iso_639-3.json@/usr/share/iso-codes/json/iso_639-3.json  --preload-file ../build/destdir/usr/share/locale@/usr/share/locale  --pr
eload-file ../build/destdir/usr/share/fcitx5@/usr/share/fcitx5  --preload-file ../build/destdir/usr/lib/fcitx5@/usr/lib/fcitx5  webpanel/libwebpanel.a  wasmfrontend/libw
asmfrontend.a  fcitx5/src/im/keyboard/libkeyboard.a  fcitx5-webview/src/libWebviewCandidateWindow.a  fcitx5/src/lib/fcitx/libFcitx5Core.so  fcitx5/src/lib/fcitx-config/l
ibFcitx5Config.so  fcitx5/src/lib/fcitx-utils/libFcitx5Utils.so  sysroot/usr/lib/libexpat.a  sysroot/usr/lib/libxkbcommon.a  sysroot/usr/lib/libjson-c.a
 /mnt/storage/github/emsdk/upstream/bin/clang --version
 /mnt/storage/github/emsdk/upstream/emscripten/tools/file_packager src/Fcitx5.data --from-emcc --preload ../build/sysroot/usr/share/X11@/usr/share/X11 ../build/sysroot/usr/share/iso-codes/json/iso_3166-1.json@/usr/share/iso-codes/json/iso_3166-1.json ../build/sysroot/usr/share/iso-codes/json/iso_639-3.json@/usr/share/iso-codes/json/iso_639-3.json ../build/destdir/usr/share/locale@/usr/share/locale ../build/destdir/usr/share/fcitx5@/usr/share/fcitx5 ../build/destdir/usr/lib/fcitx5@/usr/lib/fcitx5 --no-node
 /mnt/storage/github/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_qie0voif.rsp.utf-8
 /mnt/storage/github/emsdk/upstream/bin/wasm-ld @/tmp/emscripten_l43olzxs.rsp.utf-8
 /mnt/storage/github/emsdk/upstream/bin/llvm-objcopy src/Fcitx5.wasm src/Fcitx5.wasm --remove-section=.debug* --remove-section=producers
 /mnt/storage/github/emsdk/upstream/bin/wasm-emscripten-finalize --dyncalls-i64 --pass-arg=legalize-js-interface-export-originals --pass-arg=legalize-js-interface-exported-helpers src/Fcitx5.wasm -o src/Fcitx5.wasm --detect-features
 /mnt/storage/github/emsdk/upstream/bin/wasm-emscripten-finalize --dyncalls-i64 --pass-arg=legalize-js-interface-export-originals --pass-arg=legalize-js-interface-exported-helpers src/Fcitx5.wasm -o src/Fcitx5.wasm --detect-features
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/src/compiler.mjs /tmp/tmpzuugy7zv.json
 /mnt/storage/github/emsdk/upstream/bin/wasm-opt --strip-target-features --post-emscripten -O3 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir src/Fcitx5.wasm -o src/Fcitx5.wasm --mvp-features --enable-call-indirect-overlong --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
 /mnt/storage/github/emsdk/upstream/bin/wasm-opt --strip-target-features --post-emscripten -O3 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --no-stack-ir src/Fcitx5.wasm -o src/Fcitx5.wasm --mvp-features --enable-call-indirect-overlong --enable-multivalue --enable-mutable-globals --enable-reference-types --enable-sign-ext
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.js unsignPointers -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.js unsignPointers -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js --minify-whitespace -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso2.js
 /mnt/storage/github/emsdk/node/20.18.0_64bit/bin/node /mnt/storage/github/emsdk/upstream/emscripten/tools/acorn-optimizer.mjs /tmp/emscripten_temp_3_h7m503/Fcitx5.jso1.js --minify-whitespace -o /tmp/emscripten_temp_3_h7m503/Fcitx5.jso2.js

[kripken]

To find if this is an llvm, emscripten, or binaryen issue, link with EMCC_DEBUG=1 in the env, and view the temp files /tmp/emscripten_temp/emcc-* that are emitted. The logging output will say which tool emits which file.

[eagleoflqj]

Thanks for you instruction. There are 2 executions of wasm-ld. The second one causes difference (with same normalized response file provided). A wasm-ld -mllvm -debug @response-file logs different import order.
Definitely a lld issue, not emscripten.

[eagleoflqj]

By checking wasm-ld's log, I think the first difference comes from different emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/pic/libc.a.
On macOS arm64 it has md5sum 429f0d49ac576db7afef94c6790acce8, on Linux x64_64 it's 84ea5357d41ab10ef3075ee44234475c

[eagleoflqj]

Yeah I think it's exactly the issue. A diff of ar tv libc.a shows

1d0
< rw-r--r-- 0/0   1404 Dec 31 19:00 1969 _Exit.o
73a73
> rw-r--r-- 0/0   1404 Dec 31 19:00 1969 _exit__1.o

[eagleoflqj]

emscripten/tools/system_libs.py

Lines 232 to 243 in 6bed395

	# Resolve duplicates by appending unique.
	# This is needed on case insensitive filesystem to handle,
	# for example, _exit.o and _Exit.o.
	object_basename = shared.unsuffixed_basename(src)
	if case_insensitive:
	object_basename = object_basename.lower()
	o = os.path.join(build_dir, object_basename + '.o')
	object_uuid = 0
	# Find a unique basename
	while o in objects:
	object_uuid += 1
	o = os.path.join(build_dir, f'{object_basename}__{object_uuid}.o')

So this doesn't seem reproducible cross macOS and Linux
CC @sbc100
Does it make sense to always treat FS as case insensitive?

[sbc100]

Sure that sounds like a good idea. Would you like to send a PR?

Is macOS really case insensitive by default? I had not realized that. Strange.

[kripken]

We have an option for that (inspired by MacOS, I think?):

emscripten/src/settings.js

Lines 1013 to 1017 in bcad96d

	// If set to nonzero, the provided virtual filesystem if treated
	// case-insensitive, like Windows and macOS do. If set to 0, the VFS is
	// case-sensitive, like on Linux.
	// [link]
	var CASE_INSENSITIVE_FS = false;

Use it with emcc -sCASE_INSENSITIVE_FS. It should work in both the old FS and WasmFS.

[kripken]

Oh, wait, ignore my last comment 😆 I was mixed up by the runtime filesystem and the emsdk build.

[eagleoflqj]

Would you like to send a PR?

Yes!

Is macOS really case insensitive by default?

Yes. The is_case_insensitive function shows it is.