PR for llvm/llvm-project#61208 #347
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM |
13 tasks
rrbutani
added a commit
to rrbutani/nixpkgs
that referenced
this pull request
May 8, 2024
--- As detailed in NixOS#299849, `pkgsLLVM.libxcrypt` fails to build. At nixpkgs rev `3b61d595b17f2e3410bcc79c473060e098829eb8`: ```console $ nix-instantiate . -A libxcrypt # gcc /nix/store/finy8awd01kgg8p832m7mnm49qhm2225-libxcrypt-4.4.36.drv $ nix-instantiate . -A pkgsLLVM.libxcrypt # LLVM 17 /nix/store/wybil0v9j55681w8vwz5751j6kfaccc9-libxcrypt-x86_64-unknown-linux-gnu-4.4.36.drv ``` This is the error we get when building the latter: ```console libtool: link: x86_64-unknown-linux-gnu-clang -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ lib/.libs/libcrypt_la-alg-des.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-core.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-hmac.o \ lib/.libs/libcrypt_la-alg-hmac-sha1.o \ lib/.libs/libcrypt_la-alg-md4.o \ lib/.libs/libcrypt_la-alg-md5.o \ lib/.libs/libcrypt_la-alg-sha1.o \ lib/.libs/libcrypt_la-alg-sha256.o \ lib/.libs/libcrypt_la-alg-sha512.o \ lib/.libs/libcrypt_la-alg-yescrypt-common.o \ lib/.libs/libcrypt_la-alg-yescrypt-opt.o \ lib/.libs/libcrypt_la-crypt-bcrypt.o \ lib/.libs/libcrypt_la-crypt-des.o \ lib/.libs/libcrypt_la-crypt-gensalt-static.o \ lib/.libs/libcrypt_la-crypt-gost-yescrypt.o \ lib/.libs/libcrypt_la-crypt-md5.o \ lib/.libs/libcrypt_la-crypt-nthash.o \ lib/.libs/libcrypt_la-crypt-pbkdf1-sha1.o \ lib/.libs/libcrypt_la-crypt-scrypt.o \ lib/.libs/libcrypt_la-crypt-sha256.o \ lib/.libs/libcrypt_la-crypt-sha512.o \ lib/.libs/libcrypt_la-crypt-static.o \ lib/.libs/libcrypt_la-crypt-sunmd5.o \ lib/.libs/libcrypt_la-crypt-yescrypt.o \ lib/.libs/libcrypt_la-crypt.o \ lib/.libs/libcrypt_la-util-base64.o \ lib/.libs/libcrypt_la-util-gensalt-sha.o \ lib/.libs/libcrypt_la-util-get-random-bytes.o \ lib/.libs/libcrypt_la-util-make-failure-token.o \ lib/.libs/libcrypt_la-util-xbzero.o \ lib/.libs/libcrypt_la-util-xstrcpy.o \ -g -O2 \ -Wl,--version-script -Wl,./libcrypt.map \ -Wl,-z -Wl,defs \ -Wl,-z -Wl,text \ -Wl,-z -Wl,relro \ -Wl,-z -Wl,now \ -Wl,-soname -Wl,libcrypt.so.2 \ -o .libs/libcrypt.so.2.0.0 x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'crypt_gensalt_r' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt_r' failed: symbol not defined x86_64-unknown-linux-gnu-ld: error: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_r' failed: symbol not defined ``` Comparing the gcc build and the LLVM build: Here's the link invocation for `libcrypt.so.2.0.0` w/gcc: ```console libtool: link: gcc -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ lib/.libs/libcrypt_la-alg-des.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-core.o \ lib/.libs/libcrypt_la-alg-gost3411-2012-hmac.o \ lib/.libs/libcrypt_la-alg-hmac-sha1.o \ lib/.libs/libcrypt_la-alg-md4.o \ lib/.libs/libcrypt_la-alg-md5.o \ lib/.libs/libcrypt_la-alg-sha1.o \ lib/.libs/libcrypt_la-alg-sha256.o \ lib/.libs/libcrypt_la-alg-sha512.o \ lib/.libs/libcrypt_la-alg-yescrypt-common.o \ lib/.libs/libcrypt_la-alg-yescrypt-opt.o \ lib/.libs/libcrypt_la-crypt-bcrypt.o \ lib/.libs/libcrypt_la-crypt-des.o \ lib/.libs/libcrypt_la-crypt-gensalt-static.o \ lib/.libs/libcrypt_la-crypt-gost-yescrypt.o \ lib/.libs/libcrypt_la-crypt-md5.o \ lib/.libs/libcrypt_la-crypt-nthash.o \ lib/.libs/libcrypt_la-crypt-pbkdf1-sha1.o \ lib/.libs/libcrypt_la-crypt-scrypt.o \ lib/.libs/libcrypt_la-crypt-sha256.o \ lib/.libs/libcrypt_la-crypt-sha512.o \ lib/.libs/libcrypt_la-crypt-static.o \ lib/.libs/libcrypt_la-crypt-sunmd5.o \ lib/.libs/libcrypt_la-crypt-yescrypt.o \ lib/.libs/libcrypt_la-crypt.o \ lib/.libs/libcrypt_la-util-base64.o \ lib/.libs/libcrypt_la-util-gensalt-sha.o \ lib/.libs/libcrypt_la-util-get-random-bytes.o \ lib/.libs/libcrypt_la-util-make-failure-token.o \ lib/.libs/libcrypt_la-util-xbzero.o \ lib/.libs/libcrypt_la-util-xstrcpy.o \ -g -O2 \ -Wl,--version-script -Wl,./libcrypt.map \ -Wl,-z -Wl,defs \ -Wl,-z -Wl,text \ -Wl,-z -Wl,relro \ -Wl,-z -Wl,now \ -Wl,-soname -Wl,libcrypt.so.2 \ -o .libs/libcrypt.so.2.0.0 ``` (i.e. they're identical, modulo the compiler name: ```diff *** gcc-link-invocation 2024-05-07 07:17:46.034713173 -0700 --- clang-link-invocation 2024-05-07 07:19:38.030392378 -0700 *************** *** 1,3 **** ! libtool: link: gcc -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ --- 1,3 ---- ! libtool: link: x86_64-unknown-linux-gnu-clang -shared \ -fPIC -DPIC \ lib/.libs/libcrypt_la-alg-des-tables.o \ ``` ) `libcrypt.map` is also identical for the two builds: ```ld /* Generated from libcrypt.map.in by gen-libcrypt-map. DO NOT EDIT. */ XCRYPT_2.0 { global: crypt; crypt_gensalt; crypt_gensalt_r; crypt_gensalt_ra; crypt_gensalt_rn; crypt_r; crypt_ra; crypt_rn; xcrypt; xcrypt_gensalt; xcrypt_gensalt_r; xcrypt_r; local: *; }; XCRYPT_4.3 { global: crypt_checksalt; } XCRYPT_2.0; XCRYPT_4.4 { global: crypt_preferred_method; } XCRYPT_4.3; ``` Comparing the defined exported symbols in the `.o` files linked in across the two builds: ```bash objs=( alg-des-tables.o alg-des.o alg-gost3411-2012-core.o alg-gost3411-2012-hmac.o alg-hmac-sha1.o alg-md4.o alg-md5.o alg-sha1.o alg-sha256.o alg-sha512.o alg-yescrypt-common.o alg-yescrypt-opt.o crypt-bcrypt.o crypt-des.o crypt-gensalt-static.o crypt-gost-yescrypt.o crypt-md5.o crypt-nthash.o crypt-pbkdf1-sha1.o crypt-scrypt.o crypt-sha256.o crypt-sha512.o crypt-static.o crypt-sunmd5.o crypt-yescrypt.o crypt.o util-base64.o util-gensalt-sha.o util-get-random-bytes.o util-make-failure-token.o util-xbzero.o util-xstrcpy.o ) for build_dir in gcc llvm; do for obj in ${objs[@]}; do o=$build_dir/lib/.libs/libcrypt_la-$obj nm -C --just-symbols --defined-only --extern-only $o done > ${build_dir}_syms.txt done diff {gcc,llvm}_syms.txt ``` ... yields no differences. Here's the full list: ```txt _crypt_GOST34112012_Cleanup _crypt_GOST34112012_Final _crypt_GOST34112012_Init _crypt_GOST34112012_Update _crypt_gost_hash256 _crypt_gost_hmac256 _crypt_HMAC_SHA256_Buf _crypt_HMAC_SHA256_Final _crypt_HMAC_SHA256_Init _crypt_HMAC_SHA256_Update _crypt_PBKDF2_SHA256 _crypt_SHA256_Buf _crypt_SHA256_Final _crypt_SHA256_Init _crypt_SHA256_Update _crypt_SHA512_Buf _crypt_SHA512_Final _crypt_SHA512_Init _crypt_SHA512_Update _crypt_crypto_scrypt _crypt_yescrypt _crypt_yescrypt_decode64 _crypt_yescrypt_encode64 _crypt_yescrypt_encode_params _crypt_yescrypt_encode_params_r _crypt_yescrypt_r _crypt_yescrypt_reencrypt _crypt_yescrypt_digest_shared _crypt_yescrypt_free_local _crypt_yescrypt_free_shared _crypt_yescrypt_init_local _crypt_yescrypt_init_shared _crypt_yescrypt_kdf _crypt_crypt_bcrypt_a_rn _crypt_crypt_bcrypt_rn _crypt_crypt_bcrypt_y_rn _crypt_gensalt_bcrypt_a_rn _crypt_gensalt_bcrypt_rn _crypt_gensalt_bcrypt_y_rn _crypt_crypt_gensalt crypt_gensalt@@XCRYPT_2.0 _crypt_crypt_gost_yescrypt_rn _crypt_gensalt_gost_yescrypt_rn _crypt_crypt_scrypt_rn _crypt_gensalt_scrypt_rn _crypt_crypt_sha512crypt_rn _crypt_gensalt_sha512crypt_rn _crypt_crypt crypt@@XCRYPT_2.0 _crypt_crypt_yescrypt_rn _crypt_gensalt_yescrypt_rn crypt_checksalt@@XCRYPT_4.3 _crypt_crypt_checksalt _crypt_crypt_gensalt_ra _crypt_crypt_gensalt_rn _crypt_crypt_preferred_method _crypt_crypt_r _crypt_crypt_ra _crypt_crypt_rn crypt_gensalt_ra@@XCRYPT_2.0 crypt_gensalt_rn@@XCRYPT_2.0 crypt_preferred_method@@XCRYPT_4.4 crypt_ra@@XCRYPT_2.0 crypt_rn@@XCRYPT_2.0 crypt_r@@XCRYPT_2.0 _crypt_ascii64 _crypt_gensalt_sha_rn _crypt_get_random_bytes _crypt_make_failure_token _crypt_strcpy_or_abort ``` All of the symbols `lld` was complaining about actually are missing: ``` crypt_gensalt_r xcrypt xcrypt_gensalt xcrypt_gensalt_r xcrypt_r ``` --- Poking around a bit reveals that this is really a regression caused by the [`llvmPackages` 16 -> 17 bump on Linux](NixOS#285333); digging a little further reveals that this is ultimately caused by a change in `lld`: ```console // in the nix repl > np = import ./. {} // use gcc w/lld 16: > okay = np.libxcrypt.override { stdenv = np.stdenv.override (o: { allowedRequisites = null; cc = o.cc.override { bintools = np.llvmPackages_16.bintools; }; }); } > okay «derivation /nix/store/mdpaggjkjiw7v2qzw8g4xaf9vfw3z28d-libxcrypt-4.4.36.drv» > :b okay # builds fine, modulo some silliness with flags // use gcc w/lld *17*: > bad = np.libxcrypt.override { stdenv = np.stdenv.override (o: { allowedRequisites = null; cc = o.cc.override { bintools = np.llvmPackages_17.bintools; }; }); } > bad «derivation /nix/store/66bjyyapykl6mbmai7qbn7vwm762bdsc-libxcrypt-4.4.36.drv» > :b bad # yields the version script linker error ``` Note that with the `gcc` + `lld` 16 build above we get the following output as part of the build: ``` ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'crypt_gensalt_r' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_gensalt_r' failed: symbol not defined ld: warning: version script assignment of 'XCRYPT_2.0' to symbol 'xcrypt_r' failed: symbol not defined ``` This is because in 16 `--no-undefined-version` is enabled by default: - [commit](llvm/llvm-project@241dbd3) - [phabricator](https://reviews.llvm.org/D135402) Errors were ultimately demoted to warnings for the LLVM 16 release: - [issue](llvm/llvm-project#61208) - [PR](llvm/llvm-project-release-prs#347) - [commit](llvm/llvm-project@c1949c6) As a workaround, this PR adds in `--undefined-version` as a linker flag when building `libxcrypt` with `lld` 17 and newer. Ultimately, the "right" solution is probably for upstream to update `libcrypt.map` to not reference these symbols when they're not present in `libcrypt.so`; see: besser82/libxcrypt#181 --- Closes NixOS#299849.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves llvm/llvm-project#61208