Go binary version detection results in std::bad_alloc exception #1646
Description
We recently revamped how Pixie detects a binary's Go version (#1605). This aligned it with how the go version command implements its functionality as well as addressed other bugs in our previous implementation (as described in the linked PR) .
We received a report of a std::bad_alloc exception that occurs within this new logic, which causes the PEM to crash.
libc++abi: terminating due to uncaught exception of type std::bad_alloc: std::bad_alloc
E20230730 23:53:00.815058 10815 signal_action.cc:63] Caught Aborted, suspect faulting address 0x262c. Trace:
**************************
PC: @ 0x7f019b04c472 (unknown) abort
@ 0x562574f7f976 (unknown) abort_message
@ 0x562574f6981b (unknown) demangling_terminate_handler()
@ 0x562574f7f5f3 (unknown) std::__terminate()
@ 0x562574f81316 (unknown) __cxxabiv1::failed_throw()
@ 0x562574f812af (unknown) __cxa_throw
@ 0x562574f16384 (unknown) (anonymous namespace)::handle_oom()
@ 0x562574f868dd (unknown) tcmalloc::allocate_full_cpp_throw_oom()
@ 0x56257081c2ca (unknown) px::stirling::obj_tools::ElfReader::BinaryByteCode<>()
@ 0x56257081badb (unknown) px::stirling::obj_tools::ReadGoString()
@ 0x56257081cdde (unknown) px::stirling::obj_tools::ReadGoBuildVersion()
@ 0x562570749126 (unknown) px::stirling::GoTLSSymAddrs()
@ 0x56257073421f (unknown) px::stirling::UProbeManager::UpdateGoTLSSymAddrs()
@ 0x562570738cbf (unknown) px::stirling::UProbeManager::AttachGoTLSUProbes()
@ 0x56257073d5aa (unknown) px::stirling::UProbeManager::DeployGoUProbes()
@ 0x56257073ecec (unknown) px::stirling::UProbeManager::DeployUProbes()
@ 0x562570740d6b (unknown) _ZNSt3__114__thread_proxyB6v15006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN2px8stirling13UProbeManager22RunDeployUProbesThreadERKN4absl12lts_2022062313flat_hash_setINS7_2md4UPIDENSB_ @ 0x7f019b0aefd4 (unknown) (unknown)
@ 0x7f019b12f5bc (unknown) (unknown)
**************************
Until this bug is addressed, there are two (soon to be three) ways to remediate this issue:
--stirling_uprobe_opt_outwill be available in Vizier v0.14.11 (Add--stirling_uprobe_opt_outcli flag to allow opting binaries from uprobe attachment #1971) and will provide a mechanism for opting binaries out of uprobe instrumentation.- This is the preferred remediation once released.
- Deploy a Vizier v0.14.2 or earlier
- Set the
PX_STIRLING_DISABLE_GOLANG_TLS_TRACINGPEM flag to true (viapx deploy --pem_flags=PX_STIRLING_DISABLE_GOLANG_TLS_TRACING=true)