Runtime abort during exception handling in bolted static glibc binary

[Rin18]

I have a binary that results in a runtime crash after running bolt on it. The binary contains a simple test:

int main(int argc, char *argv[]) {
    try {
        throw 1;
    } catch (int e) {
        return 1;
    }
    return 0;
}

This was compiled with g++ test.cpp -static -o test
Then bolting the binary with llvm-bolt test -o test.bolt. Running test.out.bolt results in Aborted (core dumped) with this stack trace:

#0  0x0000000000846da8 in __pthread_kill_implementation.constprop.0 ()
#1  0x000000000083b5dc in raise ()
#2  0x0000000000800278 in abort ()
#3  0x000000000080fab8 in uw_init_context_1 ()
#4  0x000000000080ff9c in _Unwind_RaiseException ()
#5  0x0000000000801cec in __cxa_throw ()
#6  0x000000000080002c in main ()

[llvmbot]

@llvm/issue-subscribers-bolt

Author: Ash Dobrescu (Rin18)

I have a binary that results in a runtime crash after running bolt on it. The binary contains a simple test: ``` int main(int argc, char *argv[]) { try { throw 1; } catch (int e) { return 1; } return 0; } ``` This was compiled with `g++ test.cpp -static -Lglibc.static/lib -o test.out` Then bolting the binary with `llvm-bolt test.out -o test.out.bolt`. Running `test.out.bolt` results in `Aborted (core dumped)` with this stack trace: ``` #0 0x0000000000846da8 in __pthread_kill_implementation.constprop.0 () #1 0x000000000083b5dc in raise () #2 0x0000000000800278 in abort () #3 0x000000000080fab8 in uw_init_context_1 () #4 0x000000000080ff9c in _Unwind_RaiseException () #5 0x0000000000801cec in __cxa_throw () #6 0x000000000080002c in main () ```

[Rin18]

I have attached the test binary and SBOMs of the gcc compiler and glibc static library used to compile the program: binary+SBOMs.zip
Also tagging @peterwaller-arm since he discovered the crash.

[maksfb]

What is the version of GCC? Does it crash without -static? With -Wl,-q?

[Rin18]

What is the version of GCC? Does it crash without -static? With -Wl,-q?
The GCC versoin:

$g++ --version
g++ (GCC) 14.2.1 20241116

It does not crash without -static. It does still crash with -Wl,-q

[peterwaller-arm]

I spent a moment in the debugger with this, but lack the time to write it up sensibly- I'll follow up next week if needed, but wanted to share what I had noticed so far in case it is of use. The difference between the dynamic binary and the static binary seems to be in how the CFI information is loaded - they have different mechanisms.

In the static binary, the libgcc btree holding the references to the FDEs don't seem to cover the higher addresses (0x8xxxxx) of the relocated text section, which from what I could see seems to be a feature of how libgcc is computing this table. I couldn't immediately see a problem with the FDEs as they appear in the llvm-dwarfdump --eh-frame output, it doesn't look like the addresses are missing there.

I wonder if there could be an issue along the lines of libgcc being order-sensitive and therefore not ending up considering the relocated addresses.

[maksfb]

Without -Wl,-q I don't expect us to relocate code.

Another thing to try: isolate the issue and pass --funcs=main to BOLT. This will ensure that we are not modifying the runtime itself.

PS. I cannot repro locally since we don't have static runtime available and for obvious reasons I cannot run externally compiled code.

[peterwaller-arm]

--funcs=main fails. --funcs=_start succeeds.
This is indeed all with -Wl,-q.

[maksfb]

My guess is that the discovery of CFIs/eh_frame is different for the static version of the runtime. We update PT_GNU_EH_FRAME, but that might not be even present in a statically linked ELF.

[peterwaller-arm]

The Linux Standard Base Core Specification says, of CIEs (and FDEs):

If Length contains the value 0, then this CIE [ed: also FDE] shall be considered a terminator and processing shall end.

What I see in the debugger is that on startup, it goes through __register_frame_info -> get_pc_range -> classify_object_over_fdes.

Inside this function, processing of FDEs stops when it hits an FDE with Length 0. This leads to the FDEs for the relocated code being missing in the btree. If I skip that conditional branch, processing continues, and the program then continues to function as expected rather than aborting.

Looking through the bolt sources, I so far perceive that a new .eh_frame section is generated and concatenated to the original section. What I didn't yet find is any code to update the Length field of FDE of the original .eh_frame.

I wonder if there are scenarios where this could affect other unwinding code. I note that dynamic binaries don't appear to have this specific problem since they appear to use some different code for matching FDEs to PCs. The length field being zero has potential to cause other problems though.

[maksfb]

Good find.

For dynamic objects, the FDE discovery process is different as it's driven by .eh_frame_hdr/PT_GNU_EH_FRAME.

[maksfb]

I might have misunderstood the problem. Looking at BOLTed .eh_frame with llvm-readelf -u, I don't see any zero terminator CIE in the middle of the section, which is expected as we add the original .eh_frame after the new one (I thought it was the other way around), and the new one is produced without the terminator.

I suspect the static runtime has no idea where the new .eh_frame is located. We might need to redefine __EH_FRAME_BEGIN__ and update all relocations pointing to it.

https://github.com/gcc-mirror/gcc/blob/dcb9af06212e8bb36e84a1b8498c625c29abeb6f/libgcc/crtstuff.c#L260-L267

[peterwaller-arm]

Thanks for looking and thinking about this, that's interesting! I'm curious then that skipping the terminator record results in processing the relocated FDEs; if it were referencing the old .eh_frame, I wouldn't expect it to find them. That is, unless the new FDEs were situated immediately after the old ones - but then, wouldn't that imply __EH_FRAME_BEGIN__ points to the new records? There may be errors in my observations, in any case your suspicion also sounds like a promising avenue.