`force-frame-pointers` flag isn't honored always unless std is recompiled as such

[ryoqun]

The compiler flag -C force-frame-pointers=yes isn't always honored, clobbering the %rbp register and resulting in the unrecoverable stackframe for perf --call-graph fp.

It seems that if I compile rustc under -C force-frame-pointers=yes, it works as shown below:

setup

$ cargo new clobbered-frame-pointer && cd clobbered-frame-pointer

pre-built stock binary rustc

$ rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -A50 'std::sys::unix::futex::futex_wait>:' 
   Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
    Finished release [optimized] target(s) in 0.33s
000000000001fe60 <std::sys::unix::futex::futex_wait>:
   1fe60:       55                      push   %rbp
   1fe61:       41 57                   push   %r15
   1fe63:       41 56                   push   %r14
   1fe65:       41 55                   push   %r13
   1fe67:       41 54                   push   %r12
   1fe69:       53                      push   %rbx
   1fe6a:       48 83 ec 28             sub    $0x28,%rsp
   1fe6e:       41 89 f6                mov    %esi,%r14d
   1fe71:       48 89 fb                mov    %rdi,%rbx
   1fe74:       48 83 3a 00             cmpq   $0x0,(%rdx)
   1fe78:       74 4d                   je     1fec7 <std::sys::unix::futex::futex_wait+0x67>
   1fe7a:       48 8b 6a 08             mov    0x8(%rdx),%rbp
   1fe7e:       44 8b 7a 10             mov    0x10(%rdx),%r15d
   1fe82:       48 8d 74 24 10          lea    0x10(%rsp),%rsi
   1fe87:       bf 01 00 00 00          mov    $0x1,%edi
   1fe8c:       ff 15 a6 cb 02 00       callq  *0x2cba6(%rip)        # 4ca38 <clock_gettime@GLIBC_2.17>
   1fe92:       83 f8 ff                cmp    $0xffffffff,%eax
   1fe95:       0f 84 c1 00 00 00       je     1ff5c <std::sys::unix::futex::futex_wait+0xfc>
   1fe9b:       48 8b 44 24 10          mov    0x10(%rsp),%rax
   1fea0:       48 01 e8                add    %rbp,%rax
   1fea3:       0f 90 c2                seto   %dl
   1fea6:       31 c9                   xor    %ecx,%ecx
   1fea8:       48 85 ed                test   %rbp,%rbp
   1feab:       78 37                   js     1fee4 <std::sys::unix::futex::futex_wait+0x84>
   1fead:       84 d2                   test   %dl,%dl
   1feaf:       75 33                   jne    1fee4 <std::sys::unix::futex::futex_wait+0x84>
   1feb1:       48 8b 4c 24 18          mov    0x18(%rsp),%rcx
   1feb6:       41 01 cf                add    %ecx,%r15d
   1feb9:       41 81 ff 00 ca 9a 3b    cmp    $0x3b9aca00,%r15d
   1fec0:       72 10                   jb     1fed2 <std::sys::unix::futex::futex_wait+0x72>
   1fec2:       48 ff c0                inc    %rax
   1fec5:       71 04                   jno    1fecb <std::sys::unix::futex::futex_wait+0x6b>
   1fec7:       31 c9                   xor    %ecx,%ecx
   1fec9:       eb 19                   jmp    1fee4 <std::sys::unix::futex::futex_wait+0x84>
   1fecb:       41 81 c7 00 36 65 c4    add    $0xc4653600,%r15d
   1fed2:       44 89 f9                mov    %r15d,%ecx
   1fed5:       48 89 44 24 18          mov    %rax,0x18(%rsp)
   1feda:       48 89 4c 24 20          mov    %rcx,0x20(%rsp)
   1fedf:       b9 01 00 00 00          mov    $0x1,%ecx
   1fee4:       48 89 4c 24 10          mov    %rcx,0x10(%rsp)
   1fee9:       4c 8d 7c 24 18          lea    0x18(%rsp),%r15
   1feee:       4c 8b 25 a3 cd 02 00    mov    0x2cda3(%rip),%r12        # 4cc98 <syscall@GLIBC_2.2.5>
   1fef5:       4c 8b 2d ac ca 02 00    mov    0x2caac(%rip),%r13        # 4c9a8 <__errno_location@GLIBC_2.2.5>
   1fefc:       0f 1f 40 00             nopl   0x0(%rax)
   1ff00:       8b 03                   mov    (%rbx),%eax
   1ff02:       40 b5 01                mov    $0x1,%bpl                 # !!!! %RBP IS CLOBBERED
   1ff05:       44 39 f0                cmp    %r14d,%eax
   1ff08:       75 41                   jne    1ff4b <std::sys::unix::futex::futex_wait+0xeb>
   1ff0a:       4c 8b 44 24 10          mov    0x10(%rsp),%r8
   1ff0f:       4d 85 c0                test   %r8,%r8

re-built rustc under -C force-frame-pointer=rustc

 rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo +kustom-for-frame-pointer build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -A50 'std::sys::unix::futex::futex_wait>:' 
   Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
    Finished release [optimized] target(s) in 0.16s
000000000001e480 <std::sys::unix::futex::futex_wait>:
   1e480:       55                      push   %rbp
   1e481:       48 89 e5                mov    %rsp,%rbp
   1e484:       41 57                   push   %r15
   1e486:       41 56                   push   %r14
   1e488:       41 55                   push   %r13
   1e48a:       41 54                   push   %r12
   1e48c:       53                      push   %rbx
   1e48d:       48 83 ec 28             sub    $0x28,%rsp
   1e491:       48 83 3a 00             cmpq   $0x0,(%rdx)
   1e495:       41 89 f6                mov    %esi,%r14d
   1e498:       49 89 ff                mov    %rdi,%r15
   1e49b:       74 48                   je     1e4e5 <std::sys::unix::futex::futex_wait+0x65>
   1e49d:       44 8b 62 10             mov    0x10(%rdx),%r12d
   1e4a1:       48 8b 5a 08             mov    0x8(%rdx),%rbx
   1e4a5:       48 8d 75 c0             lea    -0x40(%rbp),%rsi
   1e4a9:       bf 01 00 00 00          mov    $0x1,%edi
   1e4ae:       ff 15 b4 25 03 00       callq  *0x325b4(%rip)        # 50a68 <clock_gettime@GLIBC_2.17>
   1e4b4:       83 f8 ff                cmp    $0xffffffff,%eax
   1e4b7:       0f 84 c2 00 00 00       je     1e57f <std::sys::unix::futex::futex_wait+0xff>
   1e4bd:       48 8b 45 c0             mov    -0x40(%rbp),%rax
   1e4c1:       48 01 d8                add    %rbx,%rax
   1e4c4:       0f 90 c1                seto   %cl
   1e4c7:       48 85 db                test   %rbx,%rbx
   1e4ca:       78 19                   js     1e4e5 <std::sys::unix::futex::futex_wait+0x65>
   1e4cc:       84 c9                   test   %cl,%cl
   1e4ce:       75 15                   jne    1e4e5 <std::sys::unix::futex::futex_wait+0x65>
   1e4d0:       48 8b 4d c8             mov    -0x38(%rbp),%rcx
   1e4d4:       41 01 cc                add    %ecx,%r12d
   1e4d7:       41 81 fc 00 ca 9a 3b    cmp    $0x3b9aca00,%r12d
   1e4de:       72 16                   jb     1e4f6 <std::sys::unix::futex::futex_wait+0x76>
   1e4e0:       48 ff c0                inc    %rax
   1e4e3:       71 0a                   jno    1e4ef <std::sys::unix::futex::futex_wait+0x6f>
   1e4e5:       48 c7 45 c0 00 00 00    movq   $0x0,-0x40(%rbp)
   1e4ec:       00 
   1e4ed:       eb 1a                   jmp    1e509 <std::sys::unix::futex::futex_wait+0x89>
   1e4ef:       41 81 c4 00 36 65 c4    add    $0xc4653600,%r12d
   1e4f6:       44 89 e1                mov    %r12d,%ecx
   1e4f9:       48 89 45 c8             mov    %rax,-0x38(%rbp)
   1e4fd:       48 89 4d d0             mov    %rcx,-0x30(%rbp)
   1e501:       48 c7 45 c0 01 00 00    movq   $0x1,-0x40(%rbp)
   1e508:       00 
   1e509:       4c 8b 2d 70 27 03 00    mov    0x32770(%rip),%r13        # 50c80 <syscall@GLIBC_2.2.5>
   1e510:       4c 8b 25 e9 28 03 00    mov    0x328e9(%rip),%r12        # 50e00 <__errno_location@GLIBC_2.2.5>
   1e517:       66 0f 1f 84 00 00 00    nopw   0x0(%rax,%rax,1)
   1e51e:       00 00 
   1e520:       41 8b 07                mov    (%r15),%eax
   1e523:       b3 01                   mov    $0x1,%bl           # OKAY!
   1e525:       44 39 f0                cmp    %r14d,%eax
   1e528:       75 44                   jne    1e56e <std::sys::unix::futex::futex_wait+0xee>
   1e52a:       4c 8b 45 c0             mov    -0x40(%rbp),%r8

I picked std::sys::unix::futex::futex_wait for the above example. But it seems that there's many:

$ rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release && objdump -C -d target/release/clobbered-frame-pointer | grep -E '>:$|\$0x1,%bpl' | grep -B1 bpl | head -n 50
   Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
    Finished release [optimized] target(s) in 0.33s
000000000000f7a0 <addr2line::ResUnit<R>::parse_lines>:
   11d14:       40 b5 01                mov    $0x1,%bpl
   11d47:       40 b5 01                mov    $0x1,%bpl
   11d5c:       40 b5 01                mov    $0x1,%bpl
   11d6c:       40 b5 01                mov    $0x1,%bpl
   11d7e:       40 b5 01                mov    $0x1,%bpl
--
00000000000121e0 <addr2line::ResDwarf<R>::parse>:
   1469d:       40 b5 01                mov    $0x1,%bpl
   16e6b:       40 b5 01                mov    $0x1,%bpl
   16e7d:       40 b5 01                mov    $0x1,%bpl
--
000000000001dcc0 <std::panicking::default_hook>:
   1dfc6:       40 80 f5 01             xor    $0x1,%bpl
   1e064:       40 b5 01                mov    $0x1,%bpl
   1e076:       40 b5 01                mov    $0x1,%bpl
   1e0a6:       40 b5 01                mov    $0x1,%bpl
--
000000000001fe60 <std::sys::unix::futex::futex_wait>:
   1ff02:       40 b5 01                mov    $0x1,%bpl
--
0000000000020e30 <std::backtrace_rs::symbolize::gimli::Context::new>:
   218a4:       40 b5 01                mov    $0x1,%bpl
   219a7:       40 b5 01                mov    $0x1,%bpl
   21d71:       40 b5 01                mov    $0x1,%bpl
--
0000000000022020 <std::backtrace_rs::symbolize::gimli::resolve::{{closure}}>:
   22d20:       40 80 fd 01             cmp    $0x1,%bpl
   240b1:       40 80 fd 01             cmp    $0x1,%bpl
   253bf:       40 b5 01                mov    $0x1,%bpl
   257fc:       40 f6 c5 01             test   $0x1,%bpl
   266ed:       40 b5 01                mov    $0x1,%bpl
   267d2:       40 b5 01                mov    $0x1,%bpl
--
0000000000029a50 <gimli::read::abbrev::Abbreviations::insert>:
   29b1b:       40 80 fd 01             cmp    $0x1,%bpl
--
000000000002ce30 <<rustc_demangle::v0::Ident as core::fmt::Display>::fmt>:
   2d243:       40 b5 01                mov    $0x1,%bpl
   2d2f3:       40 b5 01                mov    $0x1,%bpl
--
000000000002da20 <rustc_demangle::v0::Printer::print_backref>:
   2da99:       40 b5 01                mov    $0x1,%bpl
000000000002dbe0 <rustc_demangle::v0::Printer::print_backref>:
   2dc59:       40 b5 01                mov    $0x1,%bpl
000000000002dda0 <rustc_demangle::v0::Printer::print_backref>:
   2de19:       40 b5 01                mov    $0x1,%bpl
--
000000000002f9d0 <rustc_demangle::v0::Printer::print_type::{{closure}}>:
   2facd:       40 b5 01                mov    $0x1,%bpl

I suspect this is caused by pre-built fn machine code is statically linked into the binary, skipping compilation and not considering RUSTFLAGS.

$ rustc --version --verbose
rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52
commit-date: 2022-09-19
host: x86_64-unknown-linux-gnu
release: 1.64.0
LLVM version: 14.0.6

[the8472]

That's probably prebuilt code shipped with the standard library, have you tried using -Zbuild-std (requires nightly)?

[ryoqun]

hey, thanks for the tip. yep, i confirmed -Zbuild-std fixes the problem.

$ rm -rf target/ && RUSTFLAGS="-C force-frame-pointers=yes" cargo +nightly-2022-08-12-x86_64-unknown-linux-gnu build -Z build-std --release --target x86_64-unknown-linux-gnu && objdump -C -d $(find . -name clobbered-frame-pointer -type f) | grep -E '>:$|\$0x1,%bpl' | grep -B1 bpl | head -n 50
   Compiling compiler_builtins v0.1.78
   Compiling core v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
   Compiling libc v0.2.126
   Compiling cc v1.0.73
   Compiling memchr v2.5.0
   Compiling std v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std)
   Compiling unwind v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/unwind)
   Compiling rustc-std-workspace-core v1.99.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc)
   Compiling cfg-if v0.1.10
   Compiling adler v0.2.3
   Compiling rustc-demangle v0.1.21
   Compiling rustc-std-workspace-alloc v1.99.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-alloc)
   Compiling panic_unwind v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_unwind)
   Compiling panic_abort v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort)
   Compiling gimli v0.25.0
   Compiling std_detect v0.1.5 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/stdarch/crates/std_detect)
   Compiling miniz_oxide v0.4.0
   Compiling object v0.26.2
   Compiling hashbrown v0.12.3
   Compiling addr2line v0.16.0
   Compiling proc_macro v0.0.0 (/home/sol/.rustup/toolchains/nightly-2022-08-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/proc_macro)
   Compiling clobbered-frame-pointer v0.1.0 (/home/sol/clobbered-frame-pointer)
    Finished release [optimized] target(s) in 18.20s
$ 

[ryoqun]

so, this is a known issue? hopefully, a warn or something can be printed without being too noisy?

[the8472]

Since this also applies to other codegen options (e.g. optimization levels) almost everyone setting RUSTFLAGS would get that warning. Maybe it could be documented in the cargo book.

[alatiera]

What would be the appropriate place to raise this issue regarding building the precompiled rustc binaries that rustup distributes with frame-pointers enabled?

[workingjubilee]

I am assuming the final comment by @alatiera is essentially the "real issue" here and leaving needs-triage because it's not clear to me what the answer is, however it is of some concern to the compiler and library team, since it has to do with how std is built and shipped. Unless -Zbuild-std is accepted as the canonical option, of course, but it doesn't seem like it would be clear to users why they would need to rebuild std to support frame pointers, rather than that being done implicitly.

[alatiera]

I guess there are 2 or maybe 3 different issues here probably:

• Having frame pointers be the default in more platforms (and rustup will probably ship the default build then)
• Honoring force-frame-pointers and rebuilding the std if the existing install doesn't match
• Stabilizing -Zbuild-std maybe?

All kinda related, but also can be tackled independently I think

[saethlin]

I think there are 3 ways to address this question, with varying tradeoffs:

1. Enable frame pointers by default in distributed stdlib. This may have overhead, but we have some benchmarking so we should be able to provide some answer. This is by far the easiest option: Enable frame pointers by default #114323

2. Distribute the standard library as MIR-only rlibs, which is a theoretical idea that nobody has implemented. The now-dead tracking issue was Tracking issue for MIR-only RLIBs #38913 I am interested in this for other reasons, so I might make progress here, but pulling this off would probably be very messy. For example, just trying to blindly enable -Zalways-encode-mir produces multiple definition errors when bootstrapping, and undefined symbol errors when combined with -Zbuild-std.

3. Make -Zbuild-std work on a stable compiler. This effort has its own repo for tracking issues: https://github.com/rust-lang/wg-cargo-std-aware/issues as well as a tag here: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3A-Zbuild-std. There seems to be some activity on that repo this year which is nice, but I'm not holding my breath. There's a lot of work to do still.

[alatiera]

Enabling frame-pointers I think it's the easiest option and developers will thank you for it. There is gonna be backlash from all the people with an opinion on the internet but I think the issue is pretty clear for any person that ever profiled an application. CPython is also making frame-pointers mandatory for 3.12 last I heard.