Backtraces do not work for binaries loaded through ld.so

[jonhoo]

When a Rust binary that is executed through ld.so — the Linux dynamic linker — triggers a panic, the backtraces are useless. I've included a discussion of why executing through ld.so is valuable below, but first, to reproduce, build a simple binary that panics on execution:

$ cargo new --bin ld-so-backtraces
     Created binary (application) `ld-so-backtraces` package
$ cd ld-so-backtraces
$ echo 'fn main() { panic!() }' > src/main.rs
$ cargo b
   Compiling ld-so-backtraces v0.1.0 (/local/home/jongje/ld-so-backtraces)
    Finished dev [unoptimized + debuginfo] target(s) in 0.46s

Backtraces work fine when executing the binary directly:

$ env RUST_BACKTRACE=1 $PWD/target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   2: core::panicking::panic
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:48:5
   3: ld_so_backtraces::main
             at ./src/main.rs:1:13
   4: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

But when executing through ld.so, the debug symbols vanish:

$ /usr/lib/"ld-linux-$(uname -m).so.1" $PWD/target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
$ env RUST_BACKTRACE=1 /usr/lib/"ld-linux-$(uname -m).so.1" $PWD/target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
$ env RUST_BACKTRACE=full /usr/lib/"ld-linux-$(uname -m).so.1" $PWD/target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
stack backtrace:
   0:     0xffffab78c468 - stpcpy
   1:     0xffffab7a3b70 - tunable_list
   2:     0xffffab78a7b4 - __getcwd
   3:     0xffffab78da00 - <unknown>
   4:     0xffffab78d754 - <unknown>
   5:     0xffffab78df4c - <unknown>
   6:     0xffffab78ddf8 - <unknown>
   7:     0xffffab78c910 - getauxval
   8:     0xffffab78db80 - <unknown>
   9:     0xffffab779e60 - _dl_map_object_from_fd
  10:     0xffffab779d7c - _dl_map_object_from_fd
  11:     0xffffab77a0d0 - _dl_map_object_from_fd
  12:     0xffffab77a2e0 - open_verify.constprop.7
  13:     0xffffab77a170 - _dl_map_object_from_fd
  14:     0xffffab77a21c - _dl_map_object_from_fd
  15:     0xffffab788470 - __tunable_set_val
  16:     0xffffab77a1e8 - _dl_map_object_from_fd
  17:     0xffffab77a0fc - _dl_map_object_from_fd
  18:     0xffffab584da4 - __libc_start_main
  19:     0xffffab779fe4 - _dl_map_object_from_fd

I suspect this happens because backtraces attempt to load debug symbols through /proc/self/exe, but /proc/self/exe points to ld.so in this case, not the real binary. I don't know of any good solutions to this, but one solution is to prefer argv[0] if /proc/self/exe is ld.so. It won't be perfect (callers need to use exec -a among other things), but it'll be better than what happens today. I suspect this happens with other execution wrappers than just ld.so (does Valgrind do this for certain kinds of execution for example?), but I don't have any concrete examples.

Why ld.so?

Where I work, we execute binaries through the dynamic linker so that we can explicitly set the shared library search path to ensure that the same shared libraries used to build a given binary are used when it is executed. It has a similar effect as setting LD_LIBRARY_PATH, but with the added benefit that it does not also set the shared library search path of any programs executed down the line. There is a lot of underlying complexity here that is probably not worth digging too much into, but the idea is that a binary should ship "with its environment" (think Nix-style binaries), and so if the built Rust binary and some binary it in turn invokes are built separately, they should use different shared library search paths, namely the ones that map to exactly what they were built with. Executing through the dynamic linker achieves that, LD_LIBRARY_PATH does not. (This is also done by things like go-appimage).

Cargo has a similar issue, though its solution is going to be a little different since it can also be used a library: rust-lang/cargo#10119.

rust-analyzer also run into issues with ld.so, though from what I can tell there are fixes in the pipeline there already.

Meta

rustc --version --verbose:

rustc 1.63.0 (4b91a6ea7 2022-08-08)
binary: rustc
commit-hash: 4b91a6ea7258a947e59c6522cd5898e7c0a6a88f
commit-date: 2022-08-08
host: aarch64-unknown-linux-gnu
release: 1.63.0
LLVM version: 14.0.5

[stephenrkell]

(Thanks @jryans for pointing me at this issue.)

If I've understood, the main problem is getting the file path associated with a particular region in the program's virtual address space. A secondary problem is that the link map leaves out the executable's filename (it is recorded as the empty string), and the target of /proc/self/exe is not a correct substitute in this run-via-ld.so case.

The only reliable way I know to solve the main problem is to use /proc/self/maps or the equivalent (e.g. FreeBSD has a sysctl KERN_PROC_VMMAP). This would sidestep the secondary problem.

For example, a backtrace routine could look for the /proc/self/maps line that covers the program counter in each frame. You would still need to match that against a link map entry, to get the object's base address, but at least you would be sure of the filename. (While you can get a fair way using just dl_iterate_phdr(), dladdr() or similar, I'm not aware of a public interface that will get the 'real' executable filename out of the dynamic linker in this scenario -- if it even saves it privately anywhere, which is probably not guaranteed.)

I've grappled with this sort of thing in my liballocs and librunt projects. E.g. there is a gotcha about reading the maps file: if your program might map memory while reading it, the file might change under your feet.

[the8472]

While you can get a fair way using just dl_iterate_phdr(), dladdr() or similar, I'm not aware of a public interface that will get the 'real' executable filename out of the dynamic linker in this scenario

Would that be enough to get the debug info though? even if the file name wouldn't resolve, at least the symbols would.

[stephenrkell]

Would that be enough to get the debug info though? even if the file name wouldn't resolve, at least the symbols would.

If you don't want to call out to the filesystem, then only the dynamic linking symbols are available (.dynsym but not .symtab). Exactly which symbols that includes depends on how the binary was linked... in executables, you can get away without exporting much, so sticking with this limitation will not reliably give good backtraces.

But 'debugging information' usually doesn't refer to either of these sections, but rather to the more detailed information that is generated for a debugger's benefit (.debug_info and friends, if using DWARF). If you want reliably good backtraces then it's worth looking at that, as even .symtab isn't enough to give you a symbolic name for all code. You need to go to the filesystem to get either of those, anyway.

[pnkfelix]

@stephenrkell wrote:

The only reliable way I know to solve the main problem is to use /proc/self/maps or the equivalent (e.g. FreeBSD has a sysctl KERN_PROC_VMMAP). This would sidestep the secondary problem.

I've started looking at this approach. It looks like it may be reasonably straight-foward to implement (something on the order of 100 lines of code in a standalone prototype), and does seem to resolve the ld.so case outlined by @jonhoo .

I'm going to see if I can make the corresponding change in https://github.com/rust-lang/backtrace-rs now

[pnkfelix]

(This might be fixed, as I believe that the change from rust-lang/backtrace-rs#488 has been pulled into the current stable rust, and may just need a test. But I haven't checked carefully yet.)

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-low

[pnkfelix]

Weird:

Things seem to work here when I use the nightly build, but not when I use the 1.70 build:

% cat Cargo.toml
[package]
name = "ld-so-backtraces"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
% cat src/main.rs
fn main() { panic!() }
% rustc +1.70 --version
rustc 1.70.0 (90c541806 2023-05-31)
% rustc +nightly --version
rustc 1.72.0-nightly (2d0aa5768 2023-06-18)
% cargo +1.70 build
    Finished dev [unoptimized + debuginfo] target(s) in 0.18s
% RUST_BACKTRACE=1 /lib64/ld-linux-x86-64.so.2 ./target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
% cargo +nightly build
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
% RUST_BACKTRACE=1 /lib64/ld-linux-x86-64.so.2 ./target/debug/ld-so-backtraces
thread 'main' panicked at 'explicit panic', src/main.rs:1:13
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2d0aa57684e10f7b3d3fe740ee18d431181583ad/library/std/src/panicking.rs:593:5
   1: core::panicking::panic_fmt
             at /rustc/2d0aa57684e10f7b3d3fe740ee18d431181583ad/library/core/src/panicking.rs:67:14
   2: core::panicking::panic
             at /rustc/2d0aa57684e10f7b3d3fe740ee18d431181583ad/library/core/src/panicking.rs:117:5
   3: ld_so_backtraces::main
             at ./src/main.rs:1:13
   4: core::ops::function::FnOnce::call_once
             at /rustc/2d0aa57684e10f7b3d3fe740ee18d431181583ad/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
%

So I must need to do something deeper here than what I had expected. Looking.