Description
Current situation is the user needs to set RUSTFLAGS='-fuse-ld=lld' (iirc) at the top level crate to make linking the library and the binary work across the entire dependency graph, and using llvm-ar. Otherwise, a format error occurs.
hardened_malloc does LTO, and LTO in LLVM will produce LLVM IR bitcode that can't be linked with rustc's ld. And we don't want to have to force projects to use a RUSTFLAG that we cannot programmatically set (I've tried like 10 different ways), and would definitely create more compatibility issues.
Basically, we're doing double LTO right now. GCC static builds probably have this same issue too, but the produced IR bitcode is compatible with generic ld.
I believe a solution to this situation is we skip the LTO stage at hardened_malloc, only compile to objects using -c, use generic ar to create the static archive, and perform linking through the normal cargo linking process in build.rs.
This will require a fork of hardened_malloc with Makefile changes that our build.rs will call. Possible this could be upstreamed but I'll let someone else deal with that.
Another possible (but unlikely) solution is we use -emit-static-lib, but this may still produce LLVM IR bitcode, and would probably require additional flags to prevent linking CRT / libstdc++ using nostdlib.
Resources:
- https://www.reddit.com/r/ProgrammingLanguages/comments/vc8iph/compiling_to_llvm_ir_and_linking_to_rust_functions/
- https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-nostdlib
- https://stackoverflow.com/questions/9148890/how-to-make-clang-compile-to-llvm-ir
- https://stackoverflow.com/questions/68700716/linking-llvm-ir-libraries
- https://llvm.org/docs/CommandGuide/llc.html