Add option to not include .o files from a .a in the .rlib which uses them

[jsgf]

At the moment, if a Rust crate has a linkage with a static library (libfoo.a), it will extract all the .o files from the .a and insert them into the .rlib (which is also an ar archive).

This is fine for small libraries, but can be a problem if the library is very large, both because of the excessive IO, and from storage use.

It would be good if rustc had an option to not do this, but instead just remember a reference to the .a file for use at final binary generation time (assuming the final link has the appropriate -L native=<dir> option).

[retep998]

This is literally what specifying kind=dylib does, it just passes it on to the linker. You don't need kind=static for static libraries. The naming is terrible and now nobody will ever fix it because of backwards compatibility reasons. Just always use kind=dylib and things will generally work fine. Eventually there'll also be a kind=static-nobundle which will be the correct way to link to static libraries. Just avoid kind=static unless you specifically need that bundling behavior (which almost never happens).

[jsgf]

Ah, you mean in the .rs file, use #[(link="foo", kind="dylib")]? According to the docs the default is dynamic, so wouldn't that do what you're suggesting? (And I'd prefer the linkage type to be a compile-time option rather than something baked into the source - we use static and dynamic for different build modes anyway).

[retep998]

@jsgf Not specifying a kind defaults to dylib yes. If you specify the library to link via a build script rather than the source code, use cargo:rustc-link-lib=dylib=foo instead of cargo:rustc-link-lib=static=foo.

dylib is used to pass libraries directly to the linker, so it'll just pass -lfoo (or foo.lib for msvc), which will work fine for linking both dynamic libraries and static libraries.
static is used to bundle libraries into the rlib, which can cause problems and should be avoided unless the bundling is strictly needed for distribution.

dylib and static do not correspond to dynamic libraries and static libraries, despite their names.

[jsgf]

OK, so maybe the issue is simply that the documentation needs to be updated and/or the current names are deprecated in favour of less misleading ones.

[retep998]

cc @steveklabnik We really need to clarify this in the documentation.

[alexcrichton]

@jsgf, @retep998

I think this is a valid issue, and I wouldn't recommend using dylib as a substitute. The dylib link type has specific propagation and linkage semantics that don't end up being true for static libraries, so crates are likely to break in weird fashions if this is used incorrectly.

@retep998 I know you feel quite strongly about this topic, but we should shy away from hyperboles "The naming is terrible and now nobody will ever fix it". The best way to help fix this problem is to be part of the solution!

@jsgf for now if the I/O or perf of this is a problem, I'd probably recommend avoiding #[link] annotations entirely. That way you can delay linking these sorts of libraries until the very end when you've got a static library anyway.

In general we haven't really paid a huge amount of attention to the perf impact of #[link] and such today. Generating and reading rlibs is pretty quick, but beyond that the file formats haven't been thoroughly vetted and such.

[jsgf]

This isn't currently critical, but if rustc can't handle thin archives then it could be a real problem.

[retep998]

The dylib link type has specific propagation and linkage semantics that don't end up being true for static libraries, so crates are likely to break in weird fashions if this is used incorrectly.

The particular case where this occurs is when you are working with rust dylibs (not cdylibs), the predominant use case being compiler plugins for rustc. A static library should only be linked once into the first immediate binary (either dylib or exe) and then any reachable symbols from the static library should be exported (if it is a dylib), while a dynamic library should be linked into every binary. static and static-nobundle both handle the former case, while dylib handles the latter case. If there's only a final cdylib or exe with no rust dylibs in the mix, then this difference doesn't matter. Actually, thinking about compiler plugins further, only the plugin itself is compiled as a dylib and all its cargo dependencies are compiled as rlibs, which effectively means this difference doesn't even apply to normal usage of compiler plugins with cargo. This difference really only applies to the Rust distribution itself (because it has dylibs linked to dylibs) and people who build all their dependencies as dylibs (I have no idea how that can be done with cargo).

Although to be fair, there is one other difference that is Windows specific, which at the time of me writing this post doesn't actually exist yet, but will once a certain PR lands. Mainly, that dylib will apply dllimport on the extern symbols while static and static-nobundle won't.

tl;dr 99% of the time you can just use kind=dylib because you're not doing crazy things with Rust dylibs, and as soon as static-nobundle is finally implemented, switch to using that for static libraries so Windows users can be happy.

The best way to help fix this problem is to be part of the solution!

@alexcrichton Which is why I've been pushing for static-nobundle for so long (even opening an RFC for it!). I'm not sure what more you expect from me to "be part of the solution". Am I supposed to implement it myself?

[Mark-Simulacrum]

According to #37403, static-nobundle has been implemented and so we can close this -- that is the correct way to do this I think.