add option to use libbacktrace for backtraces in crash reports

[rainsupreme]

With this we get more detailed backtrace information, including information about static functions. Off by default - to enable you must enable at compile time:
make USE_LIBBACKTRACE=yes

before:

------ STACK TRACE ------
EIP:
./valkey-server *:6379(debugCommand+0x83)[0x571d1bd31e73]

265152 valkey-server *
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x75b431445330]
./valkey-server *:6379(debugCommand+0x83)[0x571d1bd31e73]
./valkey-server *:6379(call+0x156)[0x571d1bc761a6]
./valkey-server *:6379(processCommand+0x9ed)[0x571d1bc784ed]
./valkey-server *:6379(processInputBuffer+0x3d3)[0x571d1bca4133]
./valkey-server *:6379(readQueryFromClient+0x178)[0x571d1bca47e8]
./valkey-server *:6379(+0x22fe24)[0x571d1bdf7e24]
./valkey-server *:6379(aeMain+0x99)[0x571d1bc60f59]
./valkey-server *:6379(main+0x5a7)[0x571d1bc546f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x75b43142a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x75b43142a28b]
./valkey-server *:6379(_start+0x25)[0x571d1bc55f85]

265156 bio_rdb_save
/lib/x86_64-linux-gnu/libc.so.6(+0x98d71)[0x75b431498d71]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x20d)[0x75b43149b7ed]
./valkey-server *:6379(mutexQueuePop+0x63)[0x571d1be0f053]
./valkey-server *:6379(bioProcessBackgroundJobs+0xf7)[0x571d1bd6d9f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x75b43149caa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c)[0x75b431529c6c]

265153 bio_close_file
/lib/x86_64-linux-gnu/libc.so.6(+0x98d71)[0x75b431498d71]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x20d)[0x75b43149b7ed]
./valkey-server *:6379(mutexQueuePop+0x63)[0x571d1be0f053]
./valkey-server *:6379(bioProcessBackgroundJobs+0xf7)[0x571d1bd6d9f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x75b43149caa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c)[0x75b431529c6c]

265154 bio_aof
/lib/x86_64-linux-gnu/libc.so.6(+0x98d71)[0x75b431498d71]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x20d)[0x75b43149b7ed]
./valkey-server *:6379(mutexQueuePop+0x63)[0x571d1be0f053]
./valkey-server *:6379(bioProcessBackgroundJobs+0xf7)[0x571d1bd6d9f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x75b43149caa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c)[0x75b431529c6c]

265155 bio_lazy_free
/lib/x86_64-linux-gnu/libc.so.6(+0x98d71)[0x75b431498d71]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x20d)[0x75b43149b7ed]
./valkey-server *:6379(mutexQueuePop+0x63)[0x571d1be0f053]
./valkey-server *:6379(bioProcessBackgroundJobs+0xf7)[0x571d1bd6d9f7]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x75b43149caa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c)[0x75b431529c6c]

5/5 expected stacktraces.

------ STACK TRACE DONE ------

after:

------ STACK TRACE ------
EIP:
./valkey-server *:6379(debugCommand+0x83)[0x5a59d9850423]

257889 valkey-server *
#0 0x7e3f46845330 <unknown>
#1 0x5a59d9850423 debugCommand at /home/ubuntu/rainsupreme/src/debug.c:533
#2 0x5a59d97942f6 exitExecutionUnit at /home/ubuntu/rainsupreme/src/server.c:1368
#3 0x5a59d97942f6 call at /home/ubuntu/rainsupreme/src/server.c:3797
#4 0x5a59d979663d processCommand at /home/ubuntu/rainsupreme/src/server.c:4482
#5 0x5a59d97c2283 processCommandAndResetClient at /home/ubuntu/rainsupreme/src/networking.c:3747
#6 0x5a59d97c2283 processInputBuffer at /home/ubuntu/rainsupreme/src/networking.c:4050
#7 0x5a59d97c2938 readQueryFromClient at /home/ubuntu/rainsupreme/src/networking.c:4160
#8 0x5a59d99163d4 connDecrRefs at /home/ubuntu/rainsupreme/src/connhelpers.h:65
#9 0x5a59d99163d4 callHandler at /home/ubuntu/rainsupreme/src/connhelpers.h:80
#10 0x5a59d99163d4 connSocketEventHandler at /home/ubuntu/rainsupreme/src/socket.c:301
#11 0x5a59d977f019 aeProcessEvents at /home/ubuntu/rainsupreme/src/ae.c:488
#12 0x5a59d977f019 aeProcessEvents at /home/ubuntu/rainsupreme/src/ae.c:411
#13 0x5a59d977f019 aeMain at /home/ubuntu/rainsupreme/src/ae.c:543
#14 0x5a59d97727b7 main at /home/ubuntu/rainsupreme/src/server.c:7484
#15 0x7e3f4682a1ca __libc_start_call_main at ../sysdeps/nptl/libc_start_call_main.h:74
#16 0x7e3f4682a28b call_init at ../csu/libc-start.c:128
#17 0x7e3f4682a28b __libc_start_main_impl at ../csu/libc-start.c:347
#18 0x5a59d9774045 <unknown>

257893 bio_rdb_save
#0 0x7e3f46898d71 __futex_abstimed_wait_common64 at ./nptl/futex-internal.c:57
#1 0x7e3f46898d71 __futex_abstimed_wait_common at ./nptl/futex-internal.c:87
#2 0x7e3f46898d71 __GI___futex_abstimed_wait_cancelable64 at ./nptl/futex-internal.c:139
#3 0x7e3f4689b7ed __pthread_cond_wait_common at ./nptl/pthread_cond_wait.c:506
#4 0x7e3f4689b7ed ___pthread_cond_wait at ./nptl/pthread_cond_wait.c:627
#5 0x5a59d992d663 mutexQueueLengthInternal at /home/ubuntu/rainsupreme/src/mutexqueue.c:56
#6 0x5a59d992d663 mutexQueuePop at /home/ubuntu/rainsupreme/src/mutexqueue.c:122
#7 0x5a59d988bfa7 bioProcessBackgroundJobs at /home/ubuntu/rainsupreme/src/bio.c:255
#8 0x7e3f4689caa4 start_thread at ./nptl/pthread_create.c:447
#9 0x7e3f46929c6c clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:80

257890 bio_close_file
#0 0x7e3f46898d71 __futex_abstimed_wait_common64 at ./nptl/futex-internal.c:57
#1 0x7e3f46898d71 __futex_abstimed_wait_common at ./nptl/futex-internal.c:87
#2 0x7e3f46898d71 __GI___futex_abstimed_wait_cancelable64 at ./nptl/futex-internal.c:139
#3 0x7e3f4689b7ed __pthread_cond_wait_common at ./nptl/pthread_cond_wait.c:506
#4 0x7e3f4689b7ed ___pthread_cond_wait at ./nptl/pthread_cond_wait.c:627
#5 0x5a59d992d663 mutexQueueLengthInternal at /home/ubuntu/rainsupreme/src/mutexqueue.c:56
#6 0x5a59d992d663 mutexQueuePop at /home/ubuntu/rainsupreme/src/mutexqueue.c:122
#7 0x5a59d988bfa7 bioProcessBackgroundJobs at /home/ubuntu/rainsupreme/src/bio.c:255
#8 0x7e3f4689caa4 start_thread at ./nptl/pthread_create.c:447
#9 0x7e3f46929c6c clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:80

257891 bio_aof
#0 0x7e3f46898d71 __futex_abstimed_wait_common64 at ./nptl/futex-internal.c:57
#1 0x7e3f46898d71 __futex_abstimed_wait_common at ./nptl/futex-internal.c:87
#2 0x7e3f46898d71 __GI___futex_abstimed_wait_cancelable64 at ./nptl/futex-internal.c:139
#3 0x7e3f4689b7ed __pthread_cond_wait_common at ./nptl/pthread_cond_wait.c:506
#4 0x7e3f4689b7ed ___pthread_cond_wait at ./nptl/pthread_cond_wait.c:627
#5 0x5a59d992d663 mutexQueueLengthInternal at /home/ubuntu/rainsupreme/src/mutexqueue.c:56
#6 0x5a59d992d663 mutexQueuePop at /home/ubuntu/rainsupreme/src/mutexqueue.c:122
#7 0x5a59d988bfa7 bioProcessBackgroundJobs at /home/ubuntu/rainsupreme/src/bio.c:255
#8 0x7e3f4689caa4 start_thread at ./nptl/pthread_create.c:447
#9 0x7e3f46929c6c clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:80

257892 bio_lazy_free
#0 0x7e3f46898d71 __futex_abstimed_wait_common64 at ./nptl/futex-internal.c:57
#1 0x7e3f46898d71 __futex_abstimed_wait_common at ./nptl/futex-internal.c:87
#2 0x7e3f46898d71 __GI___futex_abstimed_wait_cancelable64 at ./nptl/futex-internal.c:139
#3 0x7e3f4689b7ed __pthread_cond_wait_common at ./nptl/pthread_cond_wait.c:506
#4 0x7e3f4689b7ed ___pthread_cond_wait at ./nptl/pthread_cond_wait.c:627
#5 0x5a59d992d663 mutexQueueLengthInternal at /home/ubuntu/rainsupreme/src/mutexqueue.c:56
#6 0x5a59d992d663 mutexQueuePop at /home/ubuntu/rainsupreme/src/mutexqueue.c:122
#7 0x5a59d988bfa7 bioProcessBackgroundJobs at /home/ubuntu/rainsupreme/src/bio.c:255
#8 0x7e3f4689caa4 start_thread at ./nptl/pthread_create.c:447
#9 0x7e3f46929c6c clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:80

5/5 expected stacktraces.

------ STACK TRACE DONE ------

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (a95a75a) to head (9a37cc6).
⚠️ Report is 3 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable   #3034       +/-   ##
============================================
- Coverage     74.71%       0   -74.72%     
============================================
  Files           129       0      -129     
  Lines         71331       0    -71331     
============================================
- Hits          53295       0    -53295     
+ Misses        18036       0    -18036     

see 129 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sarthakaggarwal97]

This looks nice. Should we explore an option of not vendoring this? The change is pretty huge.

[rainsupreme]

I suppose it's worth considering not vendoring this. How would we structure it in that case?

I somewhat fixed mac OS support, but it's more difficult to support the platform. One must run dsymutil to generate debug info, and LTO causes this to fail. It's coded to fall back to the old logic if debug info isn't found or didn't generate useful symbols.

macos fallback behavior:

------ STACK TRACE ------
EIP:
0   valkey-server                       0x000000010276dd7c debugCommand + 784

Backtrace:
libbacktrace error: executable file is not an executable (errno 0)
#0 0x1913516a4 <unknown>
#1 0x10276dd78 <unknown>
#2 0x1026c4ca4 <unknown>
#3 0x1026c7500 <unknown>
#4 0x1026e0bdc <unknown>
#5 0x1026dfe8c <unknown>
#6 0x10281b804 <unknown>
#7 0x1026abfb4 <unknown>
#8 0x1026d4c9c <unknown>
#9 0x190f76b98 <unknown>

(libbacktrace failed to resolve symbols, falling back to standard backtrace)
0   libsystem_platform.dylib            0x00000001913516a4 _sigtramp + 56
1   valkey-server                       0x000000010276dd78 debugCommand + 780
2   valkey-server                       0x00000001026c4ca4 call + 428
3   valkey-server                       0x00000001026c7500 processCommand + 3788
4   valkey-server                       0x00000001026e0bdc processInputBuffer + 712
5   valkey-server                       0x00000001026dfe8c readQueryFromClient + 160
6   valkey-server                       0x000000010281b804 connSocketEventHandler + 180
7   valkey-server                       0x00000001026abfb4 aeProcessEvents + 356
8   valkey-server                       0x00000001026d4c9c main + 20864
9   dyld                                0x0000000190f76b98 start + 6076

------ STACK TRACE DONE ------

[zuiderkwast]

I suppose it's worth considering not vendoring this. How would we structure it in that case?

I agree, it's pretty huge and entirely optional.

Let the github action jobs install it using package managers if possible. It seems to be packaged for some (most?) distros:

• https://pkgs.org/search/?q=libbacktrace
• https://repology.org/project/libbacktrace/versions

It'd also be okay to download, build and install it from source in the CI jobs. Don't add such download logic in the Makefiles though. It can be in the yaml files.

When building Valkey with USE_LIBBACKTRACE, we can require it to be installed in the system already. Check the expected locations. Possibly we could allow specifying a path to it in a make variable too if it's required.

I somewhat fixed mac OS support, but it's more difficult to support the platform.

Seems fine if it doesn't look exactly the same as in other OSes. Also fine if it's not available on all platforms.

[Copilot]

Pull request overview

This PR integrates libbacktrace as an optional dependency to produce richer, symbolized stack traces (with file/line and static functions) in crash reports, and wires it through the build system and CI.

Changes:

• Adds an optional USE_LIBBACKTRACE=yes build flag that vendored-builds and links a static libbacktrace library, defining USE_LIBBACKTRACE for the server build.
• Enhances logStackTrace and related helpers in src/debug.c to fork a helper process that uses libbacktrace to symbolize previously-captured PCs, with a fallback to backtrace_symbols_fd.
• Updates README and CI/daily/external workflows to document and exercise the new option, and adds ignore patterns for libtool artifacts.

Reviewed changes

Copilot reviewed 72 out of 79 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/debug.c	Adds libbacktrace-backed symbolization via symbolizeWithLibbacktrace, and switches stack trace printing to use it (with fallback) when USE_LIBBACKTRACE is enabled.
src/Makefile	Introduces USE_LIBBACKTRACE option, adjusting FINAL_CFLAGS, FINAL_LIBS, and dependency targets to build/link the vendored libbacktrace.
deps/Makefile	Adds libbacktrace target and includes it in distclean, so vendored libbacktrace is configured/built/cleaned as part of deps.
deps/libbacktrace/unknown.c	Part of upstream libbacktrace: default fileline resolution when file format is unknown.
deps/libbacktrace/unittest.c	Upstream libbacktrace unit test for vector allocation/release behavior.
deps/libbacktrace/ttest.c	Upstream libbacktrace multi-threaded backtrace test.
deps/libbacktrace/testlib.h	Upstream helper declarations for libbacktrace tests (structs, callbacks).
deps/libbacktrace/testlib.c	Upstream helper implementations used across libbacktrace tests.
deps/libbacktrace/test_format.c	Upstream test harness for libbacktrace’s externally visible interfaces.
deps/libbacktrace/test-driver	Upstream Automake-style test driver script used by libbacktrace tests.
deps/libbacktrace/stest.c	Upstream test for libbacktrace’s internal sort function.
deps/libbacktrace/state.c	Upstream implementation of backtrace_create_state.
deps/libbacktrace/sort.c	Upstream in-place qsort implementation used by libbacktrace.
deps/libbacktrace/simple.c	Upstream implementation of backtrace_simple using _Unwind_Backtrace.
deps/libbacktrace/read.c	Upstream non-mmap file view helper for platforms without mmap.
deps/libbacktrace/print.c	Upstream helper that prints a formatted backtrace to a FILE *.
deps/libbacktrace/posix.c	Upstream POSIX I/O helpers (backtrace_open/close).
deps/libbacktrace/nounwind.c	Upstream stub implementation when unwind library is unavailable.
deps/libbacktrace/mtest.c	Upstream minidebuginfo / build-id symbol resolution test.
deps/libbacktrace/move-if-change	Upstream helper script for conditional file moves.
deps/libbacktrace/mmapio.c	Upstream mmap-based file viewing helper.
deps/libbacktrace/mmap.c	Upstream mmap-based allocator used by libbacktrace.
deps/libbacktrace/missing	Upstream helper wrapper for potentially-missing GNU tools.
deps/libbacktrace/instrumented_alloc.c	Upstream allocation-failure-instrumented allocator for libbacktrace tests.
deps/libbacktrace/install-debuginfo-for-buildid.sh.in	Upstream helper script to install debug info by build-id for testing.
deps/libbacktrace/filetype.awk	Upstream AWK helper for detecting object file format.
deps/libbacktrace/filenames.h	Upstream header for filename/path utility macros.
deps/libbacktrace/fileline.c	Upstream core implementation for mapping PCs to file/line/function (DWARF + OS-specific discovery).
deps/libbacktrace/edtest2.c	Upstream part of allocation stress tests for libbacktrace.
deps/libbacktrace/edtest.c	Upstream allocation stress test driver for libbacktrace.
deps/libbacktrace/config/warnings.m4	Upstream Autoconf macro to configure compiler warning flags.
deps/libbacktrace/config/unwind_ipinfo.m4	Upstream macro to detect _Unwind_GetIPInfo availability.
deps/libbacktrace/config/override.m4	Upstream overrides/fixes for older Autoconf behavior.
deps/libbacktrace/config/multi.m4	Upstream multilib configuration macro.
deps/libbacktrace/config/lt~obsolete.m4	Upstream libtool compatibility/obsolete macro definitions.
deps/libbacktrace/config/ltversion.m4	Upstream libtool version metadata macro.
deps/libbacktrace/config/ltsugar.m4	Upstream libtool m4 convenience macros.
deps/libbacktrace/config/ltoptions.m4	Upstream libtool option-handling macros.
deps/libbacktrace/config/lead-dot.m4	Upstream macro to detect leading-dot filename support.
deps/libbacktrace/config/enable.m4	Upstream macro for standardized --enable-* handling.
deps/libbacktrace/config.h.in	Upstream Autoconf template header for libbacktrace configuration.
deps/libbacktrace/compile	Upstream wrapper script for compilers without -c -o.
deps/libbacktrace/backtrace.h	Upstream public libbacktrace API header.
deps/libbacktrace/backtrace.c	Upstream main implementation using _Unwind_Backtrace.
deps/libbacktrace/backtrace-supported.h.in	Upstream header template indicating libbacktrace capabilities on the platform.
deps/libbacktrace/atomic.c	Upstream fallback implementations for atomic operations.
deps/libbacktrace/allocfail.sh	Upstream shell harness for allocation-failure tests.
deps/libbacktrace/allocfail.c	Upstream allocation-failure test program.
deps/libbacktrace/alloc.c	Upstream malloc-based allocator for libbacktrace when mmap is unavailable.
deps/libbacktrace/README.md	Upstream documentation for libbacktrace usage and support matrix.
deps/libbacktrace/LICENSE	Upstream BSD-style license for libbacktrace.
deps/libbacktrace/.gitignore	Ignores libbacktrace build artifacts under the vendored directory.
deps/Makefile	Adds libbacktrace to the deps build/distclean workflow, configuring it as a static, non-shared dependency.
README.md	Documents USE_LIBBACKTRACE=yes as a build option for enhanced stack traces.
.gitignore	Adds patterns for libtool-generated *.lo and *.la files.
.github/workflows/external.yml	Builds external tests with USE_LIBBACKTRACE=yes so external server runs use symbolized stack traces.
.github/workflows/daily.yml	Updates all relevant daily test builds to compile with USE_LIBBACKTRACE=yes.
.github/workflows/coverity.yml	Runs Coverity analysis on a build that includes libbacktrace integration.
.github/workflows/codecov.yml	Runs coverage build with libbacktrace enabled to validate the integration under coverage instrumentation.
.github/workflows/ci.yml	Ensures main CI builds (including sanitizers, RDMA, TLS, 32‑bit, Alpine, etc.) exercise the libbacktrace-enabled build path.

[sarthakaggarwal97]

LGTM!

[zuiderkwast]

Idea:

      - name: Install libbacktrace
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
        with:
          repository: ianlancetaylor/libbacktrace
          ref: b9e40069c0b47a722286b94eb5231f7f05c08713
          path: libbacktrace
      - run: cd libbacktrace && ./configure && make && sudo make install
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
      - name: make
        # Fail build if there are warnings
        # build with TLS just for compilation coverage
        run: make -j4 all-with-unit-tests SERVER_CFLAGS='-Werror' BUILD_TLS=yes USE_FAST_FLOAT=yes USE_LIBBACKTRACE=yes

[rainsupreme]

oops, I didn't do this yet 👀

[zuiderkwast]

Oops, I missed that it wasn't done yet.

[zuiderkwast]

Looks like this is ready to merge. Thanks!

In the future we should probably update the actions/checkout to latest (6.0.2).

I wonder if we can avoid some of the duplication if we use matrix more, but that's too is a future improvement.