Use lld by default on x86_64-unknown-linux-gnu stable
#140525
Conversation
And remove the previous beta/stable/nightly LLD tests.
…without `-Zunstable-options`
…+]lld` - separate enabling and disabling the feature in the error - add both polarities to the dedicated test - update documentation and precondition
now that it's been stabilized, beta and stage1 need to use different flags (-C vs -Z)
rustbot
added
A-run-make
|
This PR modifies cc @jieyouxu Some changes occurred in compiler/rustc_codegen_ssa Some changes occurred in src/tools/opt-dist cc @Kobzol These commits modify compiler targets. This PR modifies If appropriate, please update |
rustbot
added
the
needs-fcp
|
Why not switch to mold instead? It's MIT licensed and generally considered faster than lld and it seems better to switch linkers once rather than twice. Also at least for now the driver should probably check whether custom linker options have been specified and continue using GNU ld if any such options are specified outside of a subset known to work on the new linker (but without explicitly selecting a linker), since I don't think either lld or mold fully support all GNU ld options including all linker script syntax. |
This PR and stabilization report is joint work with @Kobzol.
Use LLD on
x86_64-unknown-linux-gnuby default, and stabilize-Clinker-features=[+-]lldand-Clink-self-contained=[+-]linkerThis PR proposes making LLD the default linker on the
x86_64-unknown-linux-gnutarget for the artifacts we distribute, and also stabilizing the-Clinker-features=[+-]lldand-Clink-self-contained=[+-]linkercodegen options to make it possible to opt out.LLD has been used as the default linker on nightly and CI on this target since May 2024 (PR, blog post), and it seems like it is working fine, so we would like to propose stabilizing it.
The main motivation for using LLD instead of the default BFD linker is improving compilation times. For example, in the linked benchmark, it makes incremental recompilation of
ripgrepindebugmore than twice faster. Another benefit is that Rust compilation becomes more consistent and self-contained, because we will use a known version of the LLD linker, rather than "whatever GNU ld version is on the user's system".Due to the performance benefit being so huge, many people already opt into using LLD (or other fast linkers, such as mold) using various approaches (1, 2, 3, 4). By making LLD the default linker on the
x86_64-unknown-linux-gnutarget, we will be able to speed up Rust compilation out of the box, without users having to opt in or know about it.What is being stabilized
rust-lldbeing used as the default linker on thex86_64-unknown-linux-gnutarget.rust-lldis being enabled by default in the compiler artifacts distributed by our CI/rustup. It is still possible to use the system linker by default usingrust.lld = falseinbootstrap.toml, which can be useful e.g. for some Linux distros that might not want to use the LLD we distribute.-Clinker-features=[+-]lldon thex86_64-unknown-linux-gnutarget. This codegen option tells rustc to enable or disable using the LLD linker.cc) remain unstable.x86_64-unknown-linux-gnu, other targets would still need to pass-Zunstable-optionsto use it.-Clink-self-contained=[+-]linker. This codegen option tells rustc to use the self-contained linker. It's not particularly useful to turn it on by itself, but when combined with-Clinker-features=+lld, rustc will use therust-lldlinker wrapper shipped with the compiler toolchain, instead of somelldbinary that the linker driver will find in thePATH.y/yes/n/no) remain unstable.lld.To opt out of using LLD,
RUSTFLAGS="-Clinker-features=-lld"would be used. To opt out of usingrust-lld, falling back to the LLD installed on the system,RUSTFLAGS="-Clink-self-contained=-linker"would be used.Tests
When enabling
rust-lldon nightly, we also switched x64 linux to use it at stage >= 1, meaning that all tests have been running with lld since May 2024, on CI as well as contributors' machines. (Post opt-dist tests also had been using it when running their test subset earlier than that).There are also a few tests dedicated to the CLI behavior, or ensuring the default linker is indeed the one we expect:
-Clink-self-containedoptions are not inconsistent (i.e. that passing both+linkerand-linkeris an error).[+-]linkerandy/yes/n/nooptions-Clink-self-containedare stable.[+-]lldoption of-Clinker-featuresis stable.-Clinker-features=[+-]lldis only stable onx86_64-unknown-linux-gnu.-Clinker-features=+lldand-Clink-self-contained=+linker.x86_64-unknown-linux-gnuwhen the bootstraprust.lldconfig option istrue.x86_64-unknown-linux-gnuarchives actually use LLD by default.Ecosystem impact
As already stated, LLD has been used as the default linker on x64 Linux on nightly for almost a year, and we haven't seen any blockers to stabilization in that time. There were a handful of issues reported, these are discussed later below.
Furthermore, two crater runs (November 2023, February 2025), were performed to test the impact of using LLD as the default linker. A triage of the earlier crater run was previously done here, but all the important findings from both crater runs are reported below.
Below is a list of compatibility differences between BFD and LLD that we have encountered. There is a more thorough list of differences in this post from the current LLD maintainer. From that post, "99.9% pieces of software work with ld.lld without a change".
.ctors/.dtorssections#128286 reported an issue where LLD was unable to link certain CUDA library was using these sections that were using the
.ctors/.dtorsELF sections. These were deprecated a long time ago (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=46770), replaced with a more modern.init_array/.fini_arraysections. LLD doesn't (and won't) support these sections (1, 2), so if they appear in input object files, the linked artifact might produce incorrect behavior, because e.g. some global variables might not get initialized properly.However, the usage of
.ctors/.dtorsshould be very rare in practice. We have performed a crater run to test this. It has identified only 8 crates where the.ctors/.dtorssection is occurring in the final linked artifact. It was caused by a few crates using the.ctorslink section manually, and by using a very (~6 year) old version of the ctor crate.Crater run analysis
Possible workaround
It is possible to detect if
.ctors/.dtorssection is present in the final linked artifact (LLD will keep it there, but it won't be populated), and warn users about it. This check is very cheap and doesn't even appear on perf. We have benchmarked the check on a 240 MiB Chrome binary, where it took 0.8ms with page cache flushed, and 0.06ms with page cache primed (which should be the common case, as the linked artifact is written to disk just before the check is performed).In theory, this could be also solved with a linker script that moves
.ctorsto.init_array.We think that these sections should be so rare that it is not worth it to implement any workarounds for now.
Different garbage collection behavior
#130397 reported an issue where LLD (correctly) prunes a local symbol, so it is missing in the linked artifact. However, BFD keeps the same symbol, so it is a regression. This is caused by a difference in linker garbage collection.
Rust uses
--gc-sectionsand puts each function into a separate linker section, which prunes unused code. In some rare situations, code that uses so called linker encapsulation symbols might not be annotated properly, which causes it to break if--gc-sectionsis used. Note that this is essentially a bug with the object file.BFD (2.37+) uses a conservative linking mode that works around this issue, which might slightly increase binary size of the linked artifact. LLD does not use this workaround by default, but it can be made to use the conservative mode using
-z nostart-stop-gc.Perhaps more importantly, the default behavior of LLD also breaks the somewhat popular linkme crate.
To avoid this issue, we tell LLD to use the conservative mode, which maintains backwards compatibility with BFD. We found that it has no effect on compilation performance and binary size in our benchmark suite. With this change,
linkmeworks.In the future, we could stop using the conservative behavior, perhaps by emitting some future-incompat lints for some time before that.
The conservative behavior opt-in broke
cargo zigbuild, which we have already fixed.Crater run analysis
Various uncommon issues
A small number of issues that only occurred in a handful of instances were found in crater, and it is unclear if LLD is at fault or if there is some other issue that was not detected with BFD.
You can examine these here.
Missing jobserver support
LLD doesn't support the jobserver protocol for limiting the number of threads used, it simply defaults to using all available cores, and is one of the reasons why it's faster than BFD. However, this should mostly be a non-issue, because most of the linking done during high parallelism sections of
cargo buildis linking of build scripts and proc macros, which are typically very fast to link (e.g. ~50ms), and a potential oversubscription of cores thus doesn't hurt that much.When the final artifact is linked (which typically takes the most time), there should be no other sources of parallelism conflicts from compiling other code, so LLD should be able to use all available threads.
That being said, it is a difference of behavior, where previously a
-jflag was generally not using more cpu than the specified limit. It can be impactful in some resource-constrained systems, but to be clear that is already the case today due to cargo parallelism. This could be one reason to opt out of usingrust-lldon some systems.LLD has support for limiting the number of threads to use, so in theory rustc could try to get all the jobserver tokens available and use that as lld's thread limit. It'd still be suboptimal as new tokens would not be dynamically detected, and we could be using less threads than available.
We did a benchmark on a real-world crate that shows that using multiple LLD threads for intermediate artifacts doesn't seem to have a performance effect. You can find it here.
Opting out of LLD in the ecosystem
We have also examined repositories where people opted out of LLD on nightly, using this GitHub query. The summary can be found below:
Summary of LLD opt outs
Here we briefly examine the most common reasons why people use
-Zlinker-features=-lld, based on comments and git history.ld.lldNixOS/nixpkgs#314268.It's unclear whether that fixed all the Nix issues though.
History
The idea to use a faster linker by default has been on the radar for quite some time (#39915, #71515). There were very early attempts to use the gold linker by default, but these had to be reverted because of compatibility issues. Support for LLD was implemented back in 2017, but it has not been made default yet, except for some more niche targets, such as WASM, ARM Cortex or RISC-V.
It took quite some time to figure out how should the interface for selecting the linker (and the way it is invoked) look like, as it differs a lot between different platforms, linkers and compiler drivers. During that time, LLD has matured and achieved almost perfect compatibility with the default Linux linker (BFD).
#56351 stabilized
-Clinker-flavor, which is used to determine how to invoke the linker. It is especially useful on targets where selecting the linker directly with-Clinkeris not possible or is impractical.#76158 stabilized
-Clink-self-contained=[y|n], which allows overriding the compiler's heuristic for deciding whether it should use self-contained or external tools (linker, sanitizers, libc, etc.). It only allowed using the self-contained mode either for everything (y) or nothing (n), but did not allow granular choice.#85961 implemented the
-Zgcc-ldflag, which was a hacky way of opting into LLD usage.MCP 510 proposed stabilizing the behavior of
-Zgcc-ldusing more granular flags (-Clink-self-contained=linker -Clinker-flavor=gcc-lld).-Clink-self-contained=linker.-Clinker-flavorpart.-Zgcc-ld, as it was replaced by-Clinker-flavor=gnu-lld-cc+-Clink-self-contained=linker.Various linker handling refactorings were performed in the meantime: #97375, #98212, #100126, #100552, #102836, #110807, #101988, #116515
The implementation of linker flavors with LLD was causing a sort of a combinatorial explosion of various options.
#119906 suggested a different approach for linker flavors (described here), where the individual flavors could be enabled separately using
+/-(e.g.+lld).-Clinker-features(see comment 1 and comment 2), which was implemented in #123656.#124129 enabled LLD by default on nightly.
#137685, #137926 enabled the conservative gargage collection mode (
-znostart-stop-gc) to improve compatibility with BFD.#96025 (April 2022), #117684 (November 2023), #137044 (February 2025): crater runs.
Unresolved questions/concerns
-jjob limit (though I believe we have/had some open issues on sometimes using more CPU resources than the job count limit implied). As mentioned above, LLD does not support the jobserver protocol.-Clink-args=-Wl,--icf=all)..ctors/.dtorssections to provide a better error message, even if that should be rare in practice?Next steps
After the FCP completes:
Development, testing, try builds were done in #138645.
r? @petrochenkov
@rustbot label +needs-fcp +T-compiler