Skip to content

Fix libgit2 ll_find_delta leaking worker threads#568

Merged
edolstra merged 2 commits into
mainfrom
libgit2-crash
Jul 16, 2026
Merged

Fix libgit2 ll_find_delta leaking worker threads#568
edolstra merged 2 commits into
mainfrom
libgit2-crash

Conversation

@edolstra

@edolstra edolstra commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Several sentry crash reports show Nix crashing in a libgit2 pack worker thread, e.g.

SIGSEGV / SEGV_MAPERR / 0x241: Fatal Error: SIGSEGV / SEGV_MAPERR / 0x241
  libc.so.6           0x7107f72a0764 __pthread_rwlock_rdlock_full64 (pthread_rwlock_common.c:298)
  libc.so.6           0x7107f72a0764 ___pthread_rwlock_rdlock (pthread_rwlock_rdlock.c:26)
  libgit2.so.1.9      0x7107f554ee45 null
  libgit2.so.1.9      0x7107f55b2cbd null
  libgit2.so.1.9      0x7107f55bc47f null
  libgit2.so.1.9      0x7107f55bc7ae null
  libc.so.6           0x7107f729a979 start_thread (pthread_create.c:447)
  libc.so.6           0x7107f7322d2b __clone3 (clone3.S:78)

while the calling GitFileSystemObjectSinkImpl::flush() has exited or is exiting. The libgit2 addresses resolve to threaded_find_deltas, find_deltas, git_odb_read, git_cache_get_raw.

The most likely cause is that some but not all worker thread creations in libgit2's ll_find_deltas fail (we have seen other instances of thread creation failing, so that tracks). ll_find_delta then returns with an error while leaking the already created worker threads, which keep running on state that is no longer valid.

The patch to libgit2 should make it proceed with the threads that is has, i.e. it should never leak worker threads.

Also, enable debug symbols for libgit2.

Context


Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

edolstra added 2 commits July 16, 2026 12:11
If `git_thread_create` failed in `ll_find_deltas` (e.g. with EAGAIN
due to transient resource exhaustion), libgit2 returned an error while
the already-created delta-search worker threads were still running.
`GitRepoImpl::flush()` would then throw and free the packbuilder,
causing the orphaned workers to crash on freed memory (DETERMINATE-NIX-66:
SIGSEGV in `git_odb_read` → `git_cache_get_raw` → `pthread_rwlock_rdlock`
during `nix flake show`).

Patch libgit2 to proceed with however many threads were successfully
created, letting the work-stealing loop redistribute the thread-less
partitions' work (stolen in their entirety, so every object is still
searched for deltas), and to fall back to the single-threaded delta
search if no threads could be created at all. Also remove a
use-after-free of the `thread_params` array on the (unlikely)
mutex lock failure path.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
@edolstra edolstra changed the title Libgit2 crash Fix libgit2 ll_find_delta leaking worker threads Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The packaging configuration now overrides libgit2 to include debug information and a patch that handles worker thread creation failures during threaded delta search, including partial-thread and single-thread fallback paths.

Changes

libgit2 thread recovery

Layer / File(s) Summary
Threaded delta search recovery
packaging/patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch
The patch tracks successfully started workers, continues after thread creation failures, falls back to inline delta search when none start, updates work stealing for inactive partitions, and asserts mutex lock success.
libgit2 packaging override
packaging/dependencies.nix
The libgit2 override enables separate debug information and appends the pack-builder recovery patch.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is related to the libgit2 worker-thread fix, though it underspecifies the crash and thread-creation failure being addressed.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch libgit2-crash

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packaging/dependencies.nix`:
- Around line 175-180: Update the patches assignment to parenthesize the
fallback expression, using (old.patches or [ ]) before appending the libgit2
patch so the patch is included whether or not old.patches already exists.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8bd14cfe-c792-4f27-a32b-fbd84a1fd242

📥 Commits

Reviewing files that changed from the base of the PR and between 469a08e and 8a39644.

📒 Files selected for processing (2)
  • packaging/dependencies.nix
  • packaging/patches/libgit2-packbuilder-dont-fail-on-thread-create-error.patch

Comment thread packaging/dependencies.nix
@github-actions

Copy link
Copy Markdown

@github-actions
github-actions Bot temporarily deployed to pull request July 16, 2026 11:20 Inactive
@edolstra
edolstra added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit 2eca43c Jul 16, 2026
33 checks passed
@edolstra
edolstra deleted the libgit2-crash branch July 16, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants