stop exporting every symbol

[m4b]

On Linux, a typical rust ELF dynamic library will export every symbol on earth, e.g.:

objdump -T ~/projects/goblin/target/debug/libgoblin.so | wc -l
2558

which looks like most of core and std, e.g.:

107680 _ZN4core3num52_$LT$impl$u20$core..str..FromStr$u20$for$u20$u16$GT$8from_str17he60266b17184e647E (114)

Why not do this? Well, it's just The Right Thing To Do ™️ but also, technically it unnecessarily increases the size of the binary by adding dynamic symbol entries + string table entires, in addition to bloom filter words.

It might slightly increase dynamic loading time but I'm probably lying now to increase bullet points for why we shouldn't export all the symbols.

I suspect this may have come up already but I can't seem to find it, but I recall perhaps it had something to do with a requirement of std to make all their symbols public and this is somehow reflected in the binary? or perhaps it was something to do with more 3rd party linker nonsense.

Anyway, if there's a technical possibility to stop it, we should try and be good binary citizens.

[retep998]

If libgoblib.so is a dylib and not a cdylib, and you statically linked in std and friends, this means that not only is it intended to be consumed by other Rust crates, but it is also responsible for providing all of std and friends. If you don't want it providing std and friends, then you should use -Cprefer-dynamic so they're pulled in as dylib dependencies too. If you want to create something to be consumed by C, and not by Rust, then use cdylib instead which strips out all the bloat entirely.

[m4b]

@retep998 Adding cdylib seems to have no effect on my linux system (although admittedly I had no idea this distinction was made :] )

I thought perhaps it needed at least an unmangled symbol to "kick" in, but this doesn't seem to help either.

objdump -T target/debug/libdylib.so  | wc -l
2250

cat Cargo.toml 
[package]
name = "dylib"
version = "0.1.0"
authors = ["m4b <m4b.github.io@gmail.com>"]

[lib]
crate-type = ["cdylib"]

[dependencies]

cat src/lib.rs 
#[no_mangle]
pub extern fn test () {
    println!("yup");
}

[alexcrichton]

@m4b that's likely a bug in cdylibs if it's doing the wrong thing here. The intention of cdylibs are to only export a C interface, which in the past I've inspected via nm -g (which for the cdylib above has very few symbols exported). I don't know the relationship between nm -g and objdump -T, however.

[m4b]

wasn't aware of nm -g, but it seems to just be filtering all symbols in the symbol table if they are GLOBAL and FUNC.

objdump -T prints every symbol in the dynamic table entry (seems equivalent to nm -D).

nm -g unfortunately isn't sufficient, and it's really kind of lying about what the "exported" symbols are (i.e., the things other libraries can import into themselves). To test this, run strip target/debug/libdylib.so and then nm -g; it will report no symbols, because the symbol table (which is strippable) was removed. Stripping released binaries on distros is fairly common (unfortunatley, imho), as it reduces network transfer size and isn't required for dynamic linking.

This does not affect the importability of test, which is given by the symbol table referenced in the so-called _DYNAMIC array in a binary (which is what the dynamic linker accesses to see if it has a symbol another binary is referencing).

[alexcrichton]

This may be related to #32887 as well. I don't really understand most of the terms being mentioned there and how they relate to each other personally, but patches are of course always welcome!

[m4b]

I think that is related to another issue I saw with rust internal functions being assigned JUMP_SLOT relocations, which is very unusual, but didn't raise an issue (yet). I will read the above issue more thoroughly to see if it's related, but I think your suspicion that they're related might be correct. If the internal functions are given PLT entries, (and hence the JUMP_SLOT relocations), this would probably qualifiy them to be pushed into the dynamic, exported symbols array by the static linker :/

[m4b]

Motivation

So. I think we can fix this (and it is something that eventually needs fixing). To illustrate what I think we're on target for, the snappy shared object library is a good comparison because:

1. it exposes a C ABI
2. it exposes it's native language ABI (C++)

The resulting ELF ABI exactly parallels the two API headers, i.e.:

nm -D /usr/lib/libsnappy.so
0000000000005ac0 T snappy_compress
0000000000005ab0 T snappy_max_compressed_length
0000000000005a10 T snappy_uncompress
0000000000005b20 T snappy_uncompressed_length
0000000000005b40 T snappy_validate_compressed_buffer
                 U __stack_chk_fail
                 U _Unwind_Resume
                 U _ZdaPv
                 U _ZdlPv
0000000000004290 T _ZN6snappy10UncompressEPKcmPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
0000000000004990 T _ZN6snappy10UncompressEPNS_6SourceEPNS_4SinkE
0000000000004410 T _ZN6snappy11RawCompressEPKcmPcPm
0000000000004210 T _ZN6snappy13RawUncompressEPKcmPc
0000000000003d90 T _ZN6snappy13RawUncompressEPNS_6SourceEPc
...

On my distro, this binary is stripped, and nm -D libsnappy.so | wc -l is 90. If the shared object had re-exported the parts of libstdc++.so it used, it would be huge (which is what we're doing). This binary allows dynamic linking against public symbols in both it's C api, and it's C++ api, without adding a ton of symbols unrelated to it's api, which of course, are public, otherwise it couldn't link/dynamically link against them. (an example of this without an exposed C++ api is libz.so, which similarly is stripped but only exports about 40 symbols in it's dynamic symbol table).

Algo

Anyway, I've found a few places for attack, potential candidates being mostly in rustc_trans/base.rs, maybe starting around here or perhaps here.

So I'm not really familiar with the rust compiler internals at all and all the ctx and sess's and ttx's are terrifying me right now, but I think the basic algorithm is as follows:

if the crate is the base crate (i.e., the final artifact we're compiling) && it is a cdylib then:
for every symbol in final artifact:
if symbol visibility not pub && not is_extern && symbol is not defined in base crate:
llvm::LLVMSetVisibility(val, Hidden)
(or perhaps set the linkage to private, need to test)

I don't know enough about rustc internals to easily write "is the base crate" (i assume it's crate num zero, but I don't see it documented anywhere), or whether currently there is such a distinction.

I also don't see an easy way to write "is defined in base crate", as the first link I gave just ends up slurping all the symbols into a vector of reachable symbols, which is just an array of strs, but I'm sure it's somewhere; probably in one of those context references ;)

The algo might seem a little strange, but modulo me messing up booleans (which is likely), I think it's correct. Exposing a C API should not re-export all public or even extern symbols from either dynamic or statically linked dependencies. It should only export public extern definitions which we ourselves have created; hence, only public, extern symbols defined in our crate/lib. This is what libsnappy.so, libz.so and any other shared object distributed and compiled with a "normal" compiler for a C/C++ language will typically do.

The only other thing to watch out for will be the prefer-dynamic flag, which means any pub symbol from an external crate (i.e., one we didn't define), must be placed in the dynamic array (and hence be imported). The hidden flag will prevent this iirc, and we might need to instead adjust the linkage to available_externally, again, dunno, need to test.

(also, on cursory examination, this appears to not be working either? passing cargo rustc -- -C prefer-dynamic to the test crate above does not yield a binary with dynamic rust library dependencies, which I thought was it's purpose).

Anyway, yea I think it's doable, with some kind of riffing on the above algorithm.

P.S. I suspect this will remove every single one of those JUMP_SLOT relocations for the pub symbols coming from dependencies which are statically linked in, since they're now hidden, not in the dynamic array (or any symbol table), and so the linker should be free to optimize.

[michaelwoerister]

cc me

[alexcrichton]

@m4b AFAIK yeah dealing with symbol visibility (particularly hidden) is the way to tackle this. The only real hard case I think is what to do when one of the output artifacts is an rlib. If all we're producing is a cdylib, dylib, executable, or staticlib, then we can make as many symbols hidden as possible as they're all private impl details.

If we're producing an rlib, however, then this rlib will become one of a few things:

• A private implementation detail of another staticlib/dylib/executable
• A public dependency of a dylib
• A public dependency with a stable ABI (e.g. an extern interface) of a staticlib/dylib/executable

We unfortunately don't know which, we we'd have to pessimistically assume that it could become a public dependency of a dylib, causing the symbols to not be hidden...

I may not be understanding what hidden means though. We should really just play around with the options here and see what happens. If the bootstrap and tests succeed then whatever system we have should work fine.

[michaelwoerister]

FYI: I've implemented what is probably a small improvement in the right direction:
https://github.com/michaelwoerister/rust/tree/symbol-visibility

It makes sure that symbols get hidden if the only reason they are externally visible was that they are shared between object files of the same crate. You can give it a try if you build that branch.

I haven't really thought hard about all of this though. One thing to note though: We might be switching to MIR-only rlibs at some point, which would make all of this a lot easier, I suspect.

[m4b]

@michaelwoerister Your patch looks like a good start!

Unfortunately it does not seem to affect symbol visibility?

I compiled several binaries with your changes, and the only difference I noticed was that rust static libs had PC32 relocations instead PLT32 irrc.

However, strangely enough, I noticed when passing the -lto flag to compile cdylib (which doesn't seem like it should be allowed according to usage error if you try with regular dylib: error: lto can only be run for executables and static library outputs) the resulting binary with -lto is almost exactly what we want, e.g., lto and non-lto version:

target/debug/examples/rdr libgoblin.so | grep -A 3 syms
    dynsyms: ElfVec {
        count: 2288,
--
    syms: ElfVec {
        count: 69,
target/debug/examples/rdr libgoblin.so.lto | grep -A 3 syms
    dynsyms: ElfVec {
        count: 113,
--
    syms: ElfVec {
        count: 68,

Strangely though, a few functions were exported and I'm not sure why (i.e., goblin::elf::get_needed ends up being exported, but it's just an pub unsafe fn, but from_phdrs isn't exported...)

              cb50 _ZN6goblin3elf6strtab6impure59_$LT$impl$u20$goblin..elf..strtab..Strtab$LT$$u27$a$GT$$GT$6to_vec17hdbf1bbe1ad89f74dE (647)
            cde0 _ZN6goblin3elf6strtab6Strtab3get17h4e1ff1cea99b9c84E (879)
            d150 _ZN6goblin7archive6Member4size17h7103fd83ad14eda4E (26)
            d170 _ZN6goblin7archive6Member4trim17h734eb39b897ffff1E (98)
            d1e0 _ZN6goblin7archive6Member4name17h0906ce07ce1df4d7E (51)
            d220 _ZN6goblin7archive9NameIndex3get17h7ba57fa8352ea2feE (1007)
            d610 _ZN6goblin7archive7Archive3get17he1d4efb334604e52E (167)
            d6c0 wow_so_meta_doge_symbol (58)
            d700 _ZN6goblin5elf643dyn6impure10get_needed17hc46910f8ab7ca190E (503)
            d900 _ZN6goblin5elf643dyn11DynamicInfo3new17h555342b13899d36fE (2089)
            e400 _ZN6goblin5elf323dyn6impure10get_needed17h5bfa48fbc82bee6bE (508)
            e600 _ZN6goblin5elf323dyn11DynamicInfo3new17h86756cec6fd0af4aE (2070)

Regardless, whatever -lto is doing seems very very close to the resulting export table that we're after.

P.S., I suspect most of size that disappeared here was simply the dynamic string table massively shrinking since the symbols are no longer exported (another reason to get this resolved correctly):

du -h libgoblin.so.lto 
872K    libgoblin.so.lto
du -h libgoblin.so
1.6M    libgoblin.so

[michaelwoerister]

I just read up on this a little and -- if LLVM IR behaves like C here, which is likely -- we'll also need to mark the declarations as hidden, not just the definitions as my patch does. I'll update my branch to do that soonish. It might be a bit more complicated to implement though.