Dynamically link libc++ if integrating with system LLVM

[topolarity]

If we are integrating with system LLVM, build.zig attempts to statically link libc++.

As a result, LLVM ends up with a separate "copy" of the library. This causes POSIX std::error_codes generated by LLVM and Clang always compare unequal, since they refer to (duplicated) static variables.

This PR changes libc++ to be linked dynamically and adds a runtime check to verify that LLVM and Clang are using a single instance of libc++. I believe this resolves #11168 (cc @mitchellh)

I might have missed some important build system details, so feel free to take this change over if needed.

[andrewrk]

Thanks for this! Let me play around with this a bit...

[andrewrk]

On my dev machine, with LLVM compiled from source:

$ stage1/bin/zig build -p stage2 -Dskip-install-lib-files -Denable-llvm

This completes but produces a non-viable binary:

$ ldd stage2/bin/zig 
	linux-vdso.so.1 (0x00007ffd9f4e7000)
	libstdc++.so.6 => not found
	libz.so.1 => /nix/store/2yy85x1bhwmynzmpr4n29caxpfm0bkk4-zlib-1.2.12/lib/libz.so.1 (0x00007fe8dbb02000)
	libm.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libm.so.6 (0x00007fe8db9c1000)
	libpthread.so.0 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libpthread.so.0 (0x00007fe8db9a1000)
	libc.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (0x00007fe8db7cc000)
	/nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/ld-linux-x86-64.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib64/ld-linux-x86-64.so.2 (0x00007fe8dbb22000)
	libdl.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libdl.so.2 (0x00007fe8db7c5000)

Running it:

$ stage2/bin/zig build -p stage3 -Dskip-install-lib-files -Denable-llvm
stage2/bin/zig: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

I believe there are two problems here:

1. In this case we actually do want static libc++ because there are no dynamic dependencies which have a dynamic dependency against libc++. Proper logic should detect the difference between these two situations by examining the positional CLI arguments which are dynamic libraries.
2. Linking dynamically against libc++ has not been done properly, since it ended up creating a binary where the libc++ library could not be found.

I suspect that the end-game here may be to include support for proper libc++ linking directly into the compiler rather than the build system. The user would supply -lc++ and Zig would have smart defaults for whether to dynamic link it or static link it, depending on the other dynamic dependencies, if any, that are part of the compilation.

[topolarity]

Oh boy, I guess it's not called dependency hell for nothing.

Linking dynamically against libc++ has not been done properly, since it ended up creating a binary where the libc++ library could not be found.

Ah, maybe an rpath problem? Most other distros try to avoid rpath, but I think Nix uses it to version dylibs. It's easy enough to add the rpath based on the located libstdc++, but I'm not sure that'd be a good default for all distros or not.

I'm going to convert this PR to a draft until I can play around with this later this week. I want to wrestle with the linker to see if we can get it to export the statically-linked libstc++ symbols to the dynamic symbol table (-rdynamic works, but adds Clang's symbols too if it is statically linked).

I also noticed a problem with how CMake is finding llvm-config. Fix incoming shortly...

[topolarity]

Now that #12136 has landed, I'll follow up on this one shortly

[andrewrk]

Here's what it looks like building master branch now that #12136 has landed, on my nixos system:

andy@ark ~/m/zig (master)> cat ~/env/zig.nix 
with import <nixpkgs> {}; {
  tmpAoeu = stdenv.mkDerivation {
    name = "tmpAoeu";

    hardeningDisable = [ "all" ];
    buildInputs = [
      cmake
      llvmPackages_14.clang-unwrapped
      llvm_14
      lld_14
      libxml2
      zlib
    ];
  };
}

andy@ark ~/m/zig (master)> nix-shell ~/env/zig.nix

[nix-shell:~/misc/zig/build]$ cmake ..  
-- The C compiler identification is GNU 10.3.0
-- The CXX compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/myjrqkb583ak29f3r5wmxrazy33i90pi-gcc-wrapper-10.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/myjrqkb583ak29f3r5wmxrazy33i90pi-gcc-wrapper-10.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring zig version 0.10.0-dev.3326+7a09bce23
-- Found llvm: -lLLVM-14;-lrt;-ldl;-lpthread;-lm;-lz;-ltinfo;-lxml2 (Required is at least version "14") 
-- Found clang: /nix/store/fbnsh7dkx8y65wclzvqdvx1pjykrvcl3-clang-14.0.1-lib/lib/libclang-cpp.so.14 (Required is at least version "14") 
-- Found lld: /nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldMinGW.a;/nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldELF.a;/nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldCOFF.a;/nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldWasm.a;/nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldMachO.a;/nix/store/mi1qfvaljmgv6hdar7w0rx1rlmjnsb5q-lld-14.0.1-lib/lib/liblldCommon.a (Required is at least version "14") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/andy/misc/zig/build

[nix-shell:~/misc/zig/build]$ make -j12 install
[  1%] Building CXX object CMakeFiles/zigcpp.dir/src/zig_llvm.cpp.o
...
[ 98%] Linking CXX executable zig0
[ 98%] Built target zig0
[ 98%] Building self-hosted component /home/andy/misc/zig/build/zig1.o
[ 99%] Building CXX object CMakeFiles/zig.dir/src/stage1/empty.cpp.o
[100%] Linking CXX executable zig
[100%] Built target zig
Install the project...
-- Install configuration: "Debug"
-- Installing: /home/andy/misc/zig/build/stage1/bin/zig
-- Set runtime path of "/home/andy/misc/zig/build/stage1/bin/zig" to ""
-- Installing: /home/andy/misc/zig/build/stage1/lib

[nix-shell:~/misc/zig/build]$ ldd stage1/bin/zig 
	linux-vdso.so.1 (0x00007ffd01974000)
	libclang-cpp.so.14 => /nix/store/fbnsh7dkx8y65wclzvqdvx1pjykrvcl3-clang-14.0.1-lib/lib/libclang-cpp.so.14 (0x00007f3f51ce6000)
	libLLVM-14.so => /nix/store/mn1g9ypss583sci8jswaqks6ys8i5n33-llvm-14.0.1-lib/lib/libLLVM-14.so (0x00007f3f4aa81000)
	librt.so.1 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/librt.so.1 (0x00007f3f4aa76000)
	libdl.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libdl.so.2 (0x00007f3f4aa71000)
	libpthread.so.0 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libpthread.so.0 (0x00007f3f4aa4f000)
	libz.so.1 => /nix/store/2yy85x1bhwmynzmpr4n29caxpfm0bkk4-zlib-1.2.12/lib/libz.so.1 (0x00007f3f4aa31000)
	libncursesw.so.6 => /nix/store/rxq3ad21lhcbh3fw5aavw4f3h99nfnyl-ncurses-6.2/lib/libncursesw.so.6 (0x00007f3f4a9bf000)
	libxml2.so.2 => /nix/store/8n9aasc90l7g9xydaax462i9rr6pnm2d-libxml2-2.9.12/lib/libxml2.so.2 (0x00007f3f4a851000)
	libm.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libm.so.6 (0x00007f3f4a710000)
	libstdc++.so.6 => /nix/store/mc6f3fxw7zv1gshdff7wyb17kyxhymnd-gcc-10.3.0-lib/lib/libstdc++.so.6 (0x00007f3f4a53b000)
	libgcc_s.so.1 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libgcc_s.so.1 (0x00007f3f4a51f000)
	libc.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (0x00007f3f4a34a000)
	/nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/ld-linux-x86-64.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib64/ld-linux-x86-64.so.2 (0x00007f3f555eb000)
	libffi.so.8 => /nix/store/ky19kzwhxnw9klw5fp1s333fk03g14mn-libffi-3.4.2/lib/libffi.so.8 (0x00007f3f4a33d000)

[nix-shell:~/misc/zig/build]$ ls -hl stage1/bin/zig 
-rwxr-xr-x 1 andy users 80M Jul 26 12:26 stage1/bin/zig

[nix-shell:~/misc/zig/build]$ stage1/bin/zig build -p stage2 -Denable-llvm

[nix-shell:~/misc/zig/build]$ ldd stage2/bin/zig 
	linux-vdso.so.1 (0x00007ffe0f16b000)
	libclang-cpp.so.14 => not found
	libLLVM-14.so => /nix/store/mn1g9ypss583sci8jswaqks6ys8i5n33-llvm-14.0.1-lib/lib/libLLVM-14.so (0x00007f8b0c0f7000)
	libz.so.1 => /nix/store/2yy85x1bhwmynzmpr4n29caxpfm0bkk4-zlib-1.2.12/lib/libz.so.1 (0x00007f8b0c0d9000)
	libpthread.so.0 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libpthread.so.0 (0x00007f8b0c0b9000)
	libc.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (0x00007f8b0bee2000)
	/nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/ld-linux-x86-64.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib64/ld-linux-x86-64.so.2 (0x00007f8b1335e000)
	libdl.so.2 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libdl.so.2 (0x00007f8b0bedd000)
	libffi.so.8 => /nix/store/ky19kzwhxnw9klw5fp1s333fk03g14mn-libffi-3.4.2/lib/libffi.so.8 (0x00007f8b0bed0000)
	librt.so.1 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/librt.so.1 (0x00007f8b0bec5000)
	libncursesw.so.6 => /nix/store/rxq3ad21lhcbh3fw5aavw4f3h99nfnyl-ncurses-6.2/lib/libncursesw.so.6 (0x00007f8b0be51000)
	libxml2.so.2 => /nix/store/8n9aasc90l7g9xydaax462i9rr6pnm2d-libxml2-2.9.12/lib/libxml2.so.2 (0x00007f8b0bce3000)
	libstdc++.so.6 => /nix/store/mc6f3fxw7zv1gshdff7wyb17kyxhymnd-gcc-10.3.0-lib/lib/libstdc++.so.6 (0x00007f8b0bb0e000)
	libm.so.6 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libm.so.6 (0x00007f8b0b9cd000)
	libgcc_s.so.1 => /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libgcc_s.so.1 (0x00007f8b0b9b3000)

[nix-shell:~/misc/zig/build]$ stage2/bin/zig 
stage2/bin/zig: error while loading shared libraries: libclang-cpp.so.14: cannot open shared object file: No such file or directory

[topolarity]

Alright, I think this might be finally cooked:

• Moved the linkage check to zig env and zig version
• Fixed a bug in the linker's "each-lib-rpath" functionality causing missing rpaths

[andrewrk]

Thanks for this! I tested this locally and was able to do all of the following things successfully:

• build zig stage1 against system nixos llvm package
• build stage2
• build stage3
• pass all the behavior tests with stage3