libfetchers: index tarball-cache thin packs with a real ODB, stop disabling pack refresh

[jld-adriano]

Motivation

Exa CI intermittently fails Nix evaluation with a missing path inside an already-cached tarball input, e.g. on exapkgs-eval-prewarm for exa-labs/monorepo master:

error: path '«https://tarball.internal.exa.ai/github/nixos/nixpkgs/08ec31770452db15496f915f41bbf0a01fa69016.tar.gz»/pkgs/build-support/go/module.nix' does not exist

Sibling runs fail the same way on other pins («fcc13c0…»/default.nix, «be9e214…»/default.nix), on both x86_64 and aarch64 hosts. The mirror tarballs are complete and hash-valid, and when the runner caches are inspected afterwards the objects are present and git fsck --connectivity-only is clean — so this is a transient, self-healing fault in how the tarball cache is written/read, not bad upstream content. tarball-ttl is set to a year in that CI, and downloadTarball_ only re-validates a cache entry via hasObject(treeHash) on the top tree, so a tree whose descendants are not resolvable stays poisoned.

Two things in libfetchers/git-utils.cc are unsound for that setup, where up to 16 runner slots share one XDG_CACHE_HOME and therefore one tarball-cache-v2:

1. GitRepoImpl::flush() writes a thin pack (git_mempack_write_thin_pack) but indexes it with git_indexer_new(..., odb = nullptr, ...). libgit2 documents that argument as "object database from which to read base objects when fixing thin packs. Pass NULL if no thin pack is expected"; fix_thin_pack() bails with "cannot fix a thin pack without an ODB". Delta bases legitimately live in other packs here — different nixpkgs revisions share the overwhelming majority of blobs and trees, and real runner caches do contain cross-pack deltas (git verify-pack on a runner: deltas=2800 cross-base-seen=7) — so telling the indexer no thin pack is expected contradicts what the writer just produced.
2. getPool() monkey-patched refresh = nullptr onto each pooled repo's pack backend. Its comment grants that the optimization is wrong if "another process happens to be unpacking a similar tarball to the cache at the same time", dismissing it as "a very unrealistic scenario" — on this CI that is the steady state, so pooled repos can hold a stale view of the pack directory and fail to resolve objects another writer has since packed.

Context

• Pass the repository ODB (the packfilesOnly ODB, i.e. pack backend + mempack, installed via git_repository_set_odb in the constructor) to git_indexer_new, kept alive by an RAII ObjectDb through git_indexer_commit. Thin-pack bases in existing packs are now resolvable instead of being an error case the writer cannot hit only by luck.
• Drop the refresh = nullptr monkey-patch, restoring libgit2's refresh-on-miss for pooled repos. It was introduced in d1f9fe9 and moved to the pool in f7c3f02 as a syscall optimization; measured on a monorepo dev-shell eval it is noise (cold: 16.57s patched vs 16.69s baseline wall; warm mean over interleaved A/B runs: 19.554s vs 19.542s wall, 15.354s vs 15.444s Nix CPU), so correctness under a shared cache is worth more than the readdir it saved.

Honest scoping: I could not reproduce the CI failure, either sequentially or under 16-way and 32-way concurrent fetches into a shared cache, so this is not a proven fix for that specific error — it removes two unsound API usages that are the only plausible mechanisms found for it. The failure's own evidence (healthy-on-inspection caches, multiple revisions, both arches, no corruption, no cleanup job touching the cache) is consistent with a write/visibility race of this kind.

Verification: nix build .#nix-cli; nix build .#checks.x86_64-linux.{nix-fetchers-tests-run,nix-functional-tests,nix-store-tests-run}; nix flake check --no-build green (incl. pre-commit); clang-format --dry-run --Werror clean. Sequential (4 nixpkgs revisions, re-reading every earlier revision's paths from fresh processes) plus 16-way and 4×32-way concurrent shared-cache stress with the patched binary: all fetches and reads succeeded, git fsck --connectivity-only and git verify-pack -v clean with no unresolved bases.

Link to Devin session: https://app.devin.ai/sessions/4a8d0a6cdb9046d9906f0d9d22d638c3
Requested by: @jld-adriano

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.