fsmonitor: preserve proofs and avoid repeated recovery scans - #59
Open
ttaylorr-oai wants to merge 432 commits into
Open
fsmonitor: preserve proofs and avoid repeated recovery scans#59ttaylorr-oai wants to merge 432 commits into
ttaylorr-oai wants to merge 432 commits into
Conversation
In collapsed-directory mode, an untracked-cache parent may represent an entire directory by one descendant witness. If that witness becomes invalid or disappears, removing it without inspecting the directory can also hide another unvisited child that remains untracked. Compute cached validity from descendants upward after preload and invalidate collapsed ancestors when a required child proof fails. Before removing a stale collapsed witness, rescan its directory and retain the parent as untracked whenever another child survives. A focused untracked-cache test removes the cached witness while leaving a sibling present and verifies that status still reports the collapsed directory. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An external attributes file can change conversion without changing a worktree attribute manifest. Content alone is also insufficient: an ancestor or linked target can be replaced, and a missing source is safe to reuse only while its containing namespace remains stable. Capture the normalized absolute-path namespace with S07/P09 before and after observing each enabled source. For a present source, require nonblocking-open support, a regular singly linked file below the attribute-file limit, and matching descriptor, pathname, and target identities. Read the entire file into one allocation. Record source configuration and contents in one framed digest, and component and target identities in a separate namespace digest. Recheck the complete namespace for stable missing sources. Enabled sources inherit the namespace capture's fail-closed identity check; disabled sources remain unobserved and safely digestible. Reject instability rather than publishing an incomplete fingerprint. Register the fingerprint library and Clar suite in both Make and Meson. Tests separate content from metadata changes, detect an altered ancestor of a missing source, preserve disabled-source digests, and exercise both object formats. This does not select repository attribute sources or integrate fingerprints into status. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor socket is selected through the Git directory, so separate worktree paths can reach the same daemon when they share that directory. A client in the second worktree can then consume change history from a daemon that watches the first, incorrectly treating changed files in its own worktree as clean. Hash the canonical worktree path together with its device and inode, plus birth time and generation on Apple platforms. Cache the resulting 64-character SHA-256 identity in the daemon and attach it to every client query. Check the identity before interpreting the requested token; reject missing or mismatched bindings with a cookie-synchronized trivial response that forces the ordinary refresh path. The protocol change must also tolerate a daemon left running by an older Git. Such a daemon treats a bound query as an opaque token and can return a plausible trivial response. After that exact response, query an unbound capability command. If the daemon does not advertise query-v1, serialize replacement through a per-socket restart lock, stop it, and start the invoking Git executable before retrying the bound query. Keep quit, flush, and capability control commands unbound. Bound daemon lifecycle retries, and fail the query instead of trusting history when the root cannot be identified or an incompatible daemon cannot be replaced. Regression tests cover shared-gitdir worktree aliases, replacement of a legacy daemon, and acceptance of a daemon that advertises a capability superset. The replacement test also verifies that the next status neither refreshes tracked entries nor starts another daemon. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Even after every directory and ignore input has been validated, collapsed-directory traversal still reopens cached subtrees that are known to contain no untracked paths. That walk repeats work the successful preload has already established. Record recursive validation and whether each cached subtree contains untracked output. In collapsed-directory mode, skip reopening a subtree only when its directory, descendants, check-only mode, and ignore inputs remain valid and no cached untracked entry exists. Clear the recursive proof when directory or ignore state is invalidated. The untracked-cache status test verifies that an unchanged empty subtree visits no directories and that a changed descendant still falls back to traversal and reports the new untracked path. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S08 depends on S07 and S05. S05 now contains the former S01/P06 prerequisite, so join only the two independent histories before applying S08. Keep every fsmonitor test registration in sorted order in Makefile and t/meson.build. In builtin/commit.c, wt-status.c, and wt-status.h, start the attribute snapshot and UNTR preload before the token-closing refresh, and preserve each state through cleanup.
A clean-status configuration digest cannot establish which index it describes while it remains detached from the repository and index that will consume it. External attribute content and namespace must also be recorded before an index can reuse conversion-dependent history. Attach a finalized, repository-bound digest at the beginning of do_read_index(), fingerprint the system, global, and info attribute sources, and store the resulting state on the index. Ignore an unfinalized digest, another repository's digest, and a second attachment. Release the state with release_index(). Extend the existing clean-status configuration unit suite to exercise repository binding, one-shot attachment, semantic and attribute hashes, unsafe-filter state, and index-lifetime cleanup. Register the new production object with both Make and Meson. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Index attachment cannot recover the configuration seen by git status or git commit if their callbacks finish without recording it. A separate configuration pass could also bind a different stream from the one that established the commands' existing behavior. Wrap each existing status or commit callback so the original callback and clean-status digest consume the same key, value, and context. Finalize and stage the digest after the existing configuration pass and before either command reads its index. Preserve determine_whence(), advice_enabled(), the original callback, configuration order, and option handling. The index-owned attachment and its existing configuration unit coverage are supplied by S08/P01; this patch adds no command-specific regression. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An index with a null trailing checksum cannot be bound to the file that was actually read unless the platform supplies a durable file identity. Treating a directory, multiply linked file, or unsupported platform as equivalent would turn identity comparison into an unwarranted correctness guarantee. Add clean_status_identity_from_stat() for single-link regular files and make clean_status_identity_is_durable() return true only on Apple platforms. Keep unsupported platforms explicitly ineligible instead of inferring durability from stat fields alone. Register the identity object and its unit suite with Make and Meson. The tests reject directories and multiply linked files, accept a single-link regular file, and check the appropriate platform result. Actual null-checksum index verification remains a separate change. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
With index.skipHash enabled, a null trailing checksum cannot prove that verify_index_from() reopened the index that do_read_index() parsed. Replacing the pathname between those operations can otherwise make an unread index appear valid. Record the identity from the index reader's existing fstat() result. When verifying a null-checksum index on an Apple platform, compare it with the identity from the verifier's existing file observation. Reject an absent, nonregular, multiply linked, or replaced identity. Leave checksummed indexes and platforms without durable identities on their existing paths. Reuse the identity classification from S08/P03 without adding an index-read system call. Register the new object and unit suite with Make and Meson; the unit test replaces the index pathname and checks the unsupported fallback. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S05 depends on S04, all of S01 through P11, and S03. Join those histories once before applying S05. S05's final patch combines S01/P11's recursively valid empty UNTR subtrees with the paired FSUC token closed by S05. S03 and S04 add tests at the same location in t/t7527-builtin-fsmonitor.sh; keep S03's startup and worktree-binding tests before S04's invalidation tests.
A filesystem-monitor token does not establish that saved configuration, conversion rules, attribute inputs, or their complete manifest still describe the current index. Accepting duplicate, stale, or partially bound history could let status trust cached worktree state under different semantics. Recognize the FSCF index extension and delegate malformed-record rejection to the bounded clean-proof parser from S07/P07. Publish its token, configuration and semantic hashes, attribute hash, and manifest only after the complete record validates. Reject duplicate records, and adopt a manifest only when the current token, hashes, complete proof flags, and filter policy all agree. Record stronger semantic mismatches and withhold incoherent history. Integrate validation into post_read_index_from(), release all owned record and manifest storage with the index, and document the extension layout. Register the history object and unit suite with Make and Meson. A SHA-1 fixture rejects duplicate records; a SHA-256 fixture accepts coherent history and detects a changed semantic hash. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The index reader consumed an optional FSMN token and EWAH bitmap before checking their complete framing. A truncated or duplicate record could publish partial monitor state; an impossible bitmap length could allocate out of bounds or cover nonexistent index entries. Validate both FSMN versions against the extension bounds, cap version-2 tokens at 4 KiB, and check EWAH word counts, run lengths, padding, and the final running-length word. Reject a bitmap wider than a non-split index. Publish the token and bitmap only after every check succeeds, and clear all existing FSMN state on failure. Extend the read-cache helper to exercise valid records, duplicates, truncation, invalid literal and set-bit runs, nonzero padding, and an invalid final running-length-word pointer. Register the helper regression in t/t7519-status-fsmonitor.sh. Malformed optional state falls back without making the worktree appear clean. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Reading a validated FSCF record is not enough to preserve it during a generic index rewrite. Writing fresh token or stat bindings before the current provider token is revalidated would claim a semantic proof that the index has not established. Write a newly bound FSCF extension only when configuration, attributes, the complete manifest, the valid provider token, and its revalidated token all agree. Otherwise preserve an existing validated record with its token and stat bindings cleared; never serialize malformed or missing history. Add the extension to the existing index writer. Extend the history unit tests to distinguish closed proofs from preserved unbound manifests. Add a test-tool round trip and t7519 coverage that read, write, and reread a coherent FSCF record through a real index. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
FSMN identifies the token associated with tracked fsmonitor state, but the independently serialized UNTR extension cannot identify the provider boundary associated with its directory snapshot. The mere presence of both extensions cannot prove that their states agree. Define and document FSUC as a versioned optional index extension containing one NUL-terminated provider token. Register its reader with index-extension dispatch; reject empty tokens, tokens longer than 4 KiB, duplicate records, unsupported versions, truncation, and trailing data before publishing state. Provide the matching serializer and release the retained token with the index. Add a read-cache helper regression for a valid record, serializer round trip, duplicate, and truncated record. Register that helper in t/t7519-status-fsmonitor.sh. The format is independently testable; deciding when its token authenticates UNTR is a separate change. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
move_index_extensions() transfers extensions to a replacement index, but index-owned FSCF history would otherwise remain on the old state. A generic rewrite could silently discard a validated manifest, while sharing its storage would create a lifetime hazard. Copy only a parsed, valid serialized record into independently owned destination storage. Reload the saved manifest through its validated parser, copy the existing token and hashes, and leave an absent or invalid source untouched. Invoke the transfer from move_index_extensions() so ordinary index release owns each copy. Extend the existing history unit suite with a real extension transfer. Verify the copied record and manifest, invalidate the source, reject a second transfer from that source, and confirm that the independent first destination remains valid. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A well-formed FSMN bitmap and a well-formed FSUC record still do not prove that a populated untracked-cache root and tracked entries were observed at the same provider boundary. Trusting mismatched tokens can suppress the directory validation needed to detect a change. After all index extensions have been read, trust a populated untracked-cache root only when a valid on-disk FSMN token matches its FSUC token. An absent cache or root needs no token pairing. Clear the untracked proof when either extension is invalid, and write FSUC beside FSMN only when an untracked cache, a current FSMN token, and valid untracked state are present. Extend the existing read-cache parser regression to check matching and mismatched tokens and to verify that a rejected FSMN clears tracked-token validity. An invalid pair continues through ordinary untracked-cache validation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Fingerprinting an external attribute file while reading the index does not prevent the attribute parser from reopening a replaced file during preload or status collection. Cached stat data could then be evaluated with conversion rules that the original fingerprint did not cover. Capture the system, global, and info attribute bytes and namespace once and keep the immutable snapshot active from untracked-cache preload through collection. Parse snapshot lines with the ordinary attribute rules, including byte-order marks, embedded NULs, and line endings. End the snapshot and release its bounded source buffers with status. Make a failed capture or changed attribute content sticky and invalidate fsmonitor validity and the untracked cache before ordinary refresh. Preserve hook-provider behavior when semantic history is absent or only the namespace changes: hooks have no closing query and retain their reported-path contract. An observed content change still invalidates hook-derived state. Add t7531 integration coverage for file-parser parity, missing attribute history, an observed hook-time attribute change, and the hook missing-history exception. Update the existing history unit test to exercise the public strong-mismatch predicate. The namespace-only hook branch has no dedicated regression in this patch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The builtin fsmonitor client interpreted an IPC reply as unbounded C strings. A truncated token or pathname could read past the reply; an empty pathname could enter invalidation code expecting at least one byte; and a slash response could be confused with a real path. Parse the complete reply into an explicit error, delta, or trivial outcome before exposing a builtin token or path. Require a bounded builtin-prefixed token and fully terminated, nonempty, worktree- relative path records. Reserve an exact single slash for a trivial reply and retain the separate double-slash global invalidation marker. Route malformed replies through the existing scan fallback. Add unit coverage for valid paths, trivial and global responses, missing delimiters, oversized tokens, empty records, absolute paths, parent traversal, and malformed separators. Register the new unit suite in both the Makefile and t/meson.build. Hook parsing and token adoption remain unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor token can mark an entry valid even when the index has no coherent history for the configuration and attributes that determine its content. With minimal stat checks, a same-size rewrite can then be reported as clean. Rebuild the attribute manifest for expanded indexes during IPC bootstrap. Compare it with the current in-process or retained on-disk manifest, invalidate only the tracked and untracked scopes whose attribute sources changed, and preserve the last complete manifest when a rebuild fails. A legacy index with no FSCF extension is different from a mismatched proof: it contains no claim about semantic history to disprove. When it also has a valid nontrivial FSMN token with core.trustctime enabled and full core.checkStat, clear FSMN validity and seed a forward baseline through ordinary configured stat checks. This avoids hashing every tracked file solely because the index predates FSCF. The baseline still needs to finish in the bootstrap command. Preserve the freshly-proven FSMN-valid bit on entries replaced by that refresh, so that the accepted token does not defer the same migration work into the next status. Keep strong global invalidation for semantic or attribute mismatches, weak stat settings, a present FSCF without complete manifest history, provider reset or failure, manifest rebuild failure, and fresh indexes without a prior nontrivial FSMN token. Retain ordinary provider handling when reliable file identity is unavailable. The migration exception has ordinary Git stat semantics rather than a content-proof guarantee; same-size changes hidden by the platform's configured stat identity can remain hidden at that boundary. Add coverage for the forward-baseline lane, the weak-stat same-size rewrite, and the refreshed baseline FSMN bits, along with unit coverage for coherent, manifest-only, missing, and present-without-manifest history. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A validated builtin daemon response already contains complete, nonempty, NUL-terminated path records. Sending those records through the hook-oriented byte-by-byte offset scanner repeats framing work and obscures the distinction between builtin and hook protocols. Introduce apply_fsmonitor_paths() and call it immediately from the builtin branch of refresh_fsmonitor(). Walk the already validated path buffer, invalidate each reported path exactly once, and retain the resulting path count. Preserve hook token offsets, malformed-response handling, and global invalidation. The parser and unit tests from S05/P04 provide the bounded input; this refactor adds no separate benchmark or test execution claim. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An exact clean-result sidecar must remain bound to one physical index, but resumable fsmonitor history needs to survive a format-only rewrite by another Git implementation. It cannot use the index checksum or file identity as its cross-implementation key. Promote the checksummed path snapshot operations needed by an external store. They open the named index without following its final symlink and retain the descriptor, then require the descriptor and current pathname to identify the same valid index. Null checksums remain ineligible for durable snapshot pins. When fstat identity is reliable, retain the validated reader descriptor for process-local proof epochs only; generic certification and persisted CSHS still require a non-null checksum. Define a canonical digest of the ordered logical entries. Include the entry count and each path, stage, object ID, mode, CE_VALID, skip-worktree, and intent-to-add state, while excluding index encoding, cached stat data, and acceleration-only flags. Unsupported transient state rejects the digest rather than disappearing with the process. Add the checksummed CSHS codec and a local-APFS-only, nofollow, atomically-replaced store bounded to eight 16-MiB namespace slots. This commit has no status caller; the following history patch restores and saves complete checkpoints through this persistence layer. Cover both object formats, malformed and null-checksum snapshots, pathname replacement, logical-entry bindings, malformed and independent checkpoint namespaces, bounded retention, and idempotent writes. A checkpoint may contain only the required FSMN and FSCF payloads. Skip absent optional payloads rather than handing a NULL source and zero length to memcpy(). Extend the malformed-checkpoint unit test to round-trip that minimal valid form before its rejection cases. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A version-2 fsmonitor hook can return consecutive NUL delimiters after its token. The hook parser passed the resulting empty record to pathname invalidation, whose callback inspects the last byte of a nonempty path. Skip zero-length hook records and count only pathnames that actually reach fsmonitor_refresh_callback(). Preserve valid reported paths, the existing treatment of a final unterminated hook record, and the separately validated builtin response path. Add a t/t7519-status-fsmonitor.sh regression whose hook emits an empty record before a modified tracked path. Require status to report the real modification without processing the empty pathname. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A collapsed sparse index cannot enumerate every tracked path needed for a complete attribute manifest. Expanding the live index would discard the sparse representation that status is supposed to preserve. Pin the named index with S09/P02, reread the verified index into a scratch index, and expand only that scratch copy. Build the complete manifest from the expanded scratch index. Check that both the parsed scratch state and original live state still match the held descriptor and stored trailer checksum; discard the manifest if either check fails. Add a sparse-checkout regression that checks the collapsed outside entry before and after status while detecting a same-size tracked rewrite. Extend the existing index unit case with a parsed A-to-B-to-A mismatch. Failed snapshot validation retains ordinary full-invalidation fallback. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The fsmonitor failure path cleared tracked validity bits and disabled untracked-cache monitoring, but left the separate fsmonitor_untracked_valid proof intact. An untrusted cache token could therefore outlive the tracked state it was meant to certify. Extract invalidate_all_fsmonitor() and call it from the existing failure branch. Clear every CE_FSMONITOR_VALID bit, revoke the untracked-token proof, disable fsmonitor use for the untracked cache, and set FSMONITOR_CHANGED only when a tracked validity bit actually changed. The new helper has an immediate production consumer. It neither issues nor closes a provider token and introduces no independent benchmark. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean provider response closes only the filesystem interval after its starting token. It cannot certify a refresh that started before the named index, configuration, attributes, and manifest were captured, or one whose semantic inputs subsequently changed. Capture the proof epoch before each refresh whose provider token may be accepted. Pin the named index, starting token, repository configuration, external attribute fingerprint, and complete full-index manifest. Recheck those inputs after the closing query. Record semantic history only for the accepted token; reject missing or changed inputs and fall back to a complete refresh. For a null-checksum index, let only the proof-epoch pin use the process-local reader descriptor retained by the preceding patch. The proof-only exception rechecks both the retained source descriptor's original stat identity and the current named path when pinning and closing the epoch. Generic certification and persisted CSHS continue to reject the null trailer. Always rebuild the manifest when physical history is unavailable, even if the stored semantic configuration already matches. Without that manifest, a trivial response invalidates the old binding and leaves the closing query with no complete epoch to bind, so each later status repeats the fallback. Teach this lifecycle to restore and save complete external history checkpoints through the preceding CSHS store. A restore validates the logical index and all FSMN, UNTR/FSUC, and FSCF sections in scratch state, then rechecks the pinned index before installing them together. A save requires the same logical entries before and after status and a closed, writable proof. Keep both paths dormant until a later patch enables them only for a normal top-level status. A retry inside a captured epoch can also lose a freshly acquired CE_FSMONITOR_VALID bit when replace_index_entry() applies its generic conservative invalidation. Mark proof-epoch refreshes explicitly and restore only a validity bit acquired by the replacement itself. Changed or rejected closures still invalidate those provisional bits before falling back. Register clean-status-epoch.c in Make and Meson alongside its first production consumer in wt-status.c. Add scripted regressions for capture-before-refresh ordering and recovery from unbound physical history. Add unit coverage for the complete full-index manifest, the restricted post-status logical-digest exception, retained-descriptor lifetime, a stat-visible same-inode size change, and atomic path replacement. Later activation patches cover external checkpoint recovery and the immediate warm run. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Untracked status used one helper both to traverse the worktree and to copy untracked and ignored entries into status output. Checking whether a traversal actually validated the repository's UNTR cache requires that directory walk without copying results or recording user-facing timing. Factor the walk into wt_status_collect_untracked_1() with an explicit collection flag. Return whether the traversal used the index's own untracked cache, and populate the result lists and advice timing only when collection is requested. Retain wt_status_collect_untracked() as the collecting wrapper. Every existing production caller still requests collection, so status output and ordinary traversal behavior remain unchanged. Token adoption and validation-only production use are not added by this preparatory patch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider event can change a tracked .gitattributes file after status captures its conversion inputs. Accepting the resulting token against the previous manifest can incorrectly reuse tracked validity when a complete status would report a content change. Record when provider invalidation expires the current manifest and semantic proof. Before retrying token closure, rebuild that manifest and recapture external attribute sources when their content changes. Keep the response token pending until the new scan and current attribute epoch are both closed. Preserve ordinary provider handling when file identity is unreliable. Preserve reusable manifest history across ordinary index rewrites without retaining expired bindings. Extend the history and manifest unit cases and the index round-trip helper. Add a scripted regression for tracked attribute changes. Manifest or snapshot failure still forces a complete scan. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider token obtained before a tracked or untracked scan does not cover worktree changes racing with that scan. Publishing it as an FSMN or FSUC proof can make a later status trust an index or untracked-cache snapshot that was never valid at that boundary. Keep bootstrap tokens pending while tracked entries are refreshed and any rooted untracked cache is traversed. For builtin providers, query again after the scans, apply intervening paths, and repeat the affected scans until a clean boundary is found or three closing queries are exhausted. Treat a trivial closing reply as complete invalidation followed by another scan; accept its replacement token only after a later clean reply. Reject provider errors, incomplete cache proofs, and exhausted retries with strong invalidation and complete fallback scans. Hook providers cannot perform a closing IPC query, so accept their token only after a complete tracked and applicable untracked collection; reject failed or trivial hook replies. A matching on-disk FSUC token can now authorize replay of recursive UNTR validity established by S01. Reconstruct that validity only after the entire extension has decoded, and only for directories without a cached per-directory exclude digest. This lets a warm status prune known-empty subtrees while still rechecking a changed .gitignore, including changes made through an unwatched hardlink alias. Trust an indexed exclude's metadata alone only when its identity is reliable and it has exactly one link; otherwise retain the complete content-hash check. Route both status collection and commit index refresh through the shared closure. Preserve ordinary behavior for existing paired state, path-limited requests, and ignored-mode collection. Cover clean and changed closures, trivial replies, retry exhaustion, provider errors, on-disk FSMN/FSUC publication, warm empty-subtree pruning, descendant events, and cached exclude changes. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Ordinary index preload assigns existing paths directly to workers. A physical directory walk instead discovers new tasks while it runs, so an unbounded queue can exhaust descriptors or strand tasks when worker creation fails. Add a directory-task queue that retains parent and child identities, budgets descriptors against RLIMIT_NOFILE, and tracks queued as well as in-flight work. Reserve at most 128 task descriptors, leave up to 16 for the rest of the process, and run a worker synchronously when extra threads cannot start. Register the common queue for Darwin in Make, CMake, and Meson. No bulk scan is invoked from preload_index(), so existing behavior is unchanged. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A bulk directory worker must locate each observed tracked path and decide whether a directory has tracked descendants. Plain pathname ordering cannot answer either question correctly for sparse indexes. Add sparse-aware entry and descendant lookups with unseen, clean, content-check, and fallback states. Compare observed metadata with ie_match_stat(), and make duplicate observations fall back through an atomic compare-and-exchange or the existing queue mutex. Skip staged, intent-to-add, skip-worktree, removed, and otherwise ineligible entries. Register the index classifier in Make, CMake, and Meson without introducing deletion outcomes or content proofs. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
4e82cbf (fsmonitor: retain suspended history across backoff writers, 2026-08-17) rejects suspended history whenever GIT_INDEX_FILE is set. But run_commit_hook() exports that variable even for an as-is commit using the main index. A clean add --refresh in such a hook rewrites the index and discards FSMN and FSUC. Repeated staging loses the historical boundary needed for later recovery as well. Authenticate the selected index against the worktree-specific gitdir's physical index, independently of the environment-selected index path. Accept normalized names only when they identify the same singly linked regular file, and bind the existing proof-epoch descriptor to that file. Carry the actual lock destination into pending-FSUC serialization so a private output cannot inherit this exception. Reuse the same check for the optional cache-tree write. Live-IPC admission is unchanged. Suspended history still needs revalidation before publishing a full proof. Exercise genuine primary and linked-worktree hooks, normalized canonical names, repeated staging, and an invalid cache tree. Seed the temporary indexes used by commit -a and partial commits with an authenticated proof and require them to reject it. Also check copied, symlinked, and hardlinked index aliases without letting the probes rewrite the main index.
4e82cbf (fsmonitor: retain suspended history across backoff writers, 2026-08-17) preserves authenticated index history while a watch-limit marker temporarily disables the provider. The apply entrypoints still enable that history only for a live IPC provider, and the patch writer requires a currently paired token. Consequently, even a same-path patch discards FSMN and FSUC. In git am, the initial index refresh can discard them before the patch changes any staged content. Attach history before the first index read during authenticated backoff. Let the existing same-path regular-file replacement path accept a suspended semantic baseline with pending untracked candidates. Keep the live-token checks unchanged, and retain the whole-patch-list preflight so any structural or otherwise unsafe patch revokes preservation for the batch. The index writer still emits only historical FSCF flags and an all-dirty tracked bitmap until a fresh provider boundary closes. Cover consecutive apply --index and am operations, a mixed structural batch, immediate index and tree checks, and recovery through a genuine TRIVIAL response followed by a successful closure.
0d366ab (fsmonitor: recognize the main index in commit hooks, 2026-08-17) keeps temporary indexes from admitting main-index history. That distinction is necessary, but a normal commit passes its real index lock to pre-commit and publishes that same file on success. A hook which refreshes the temporary index can therefore discard the suspended history which the parent was about to publish. Capture a historical-only checkpoint while the parent owns the main index lock, and seal the entries actually written before running any post-index-change hook. At successful publication, restore the history only if the hook's final entries, canonical source, configuration, and attribute inputs still match. Rewrite the pinned hook output through a nested lock and the normal index serializer, preserving its stat data and the usual racy-entry handling. The restored tracked bitmap remains entirely dirty, and the untracked cache still requires revalidation. No current clean proof is granted. Partial commits and hooks which change entries or semantic inputs keep their own output unchanged. Cover successful no-op and refresh hooks, partial commits, and real hook mutations in both worktree layouts.
The successful-hook fixture deliberately starts with staged changes to sibling. Its removal hook must therefore pass -f to git rm; otherwise Git correctly refuses the removal before the publication assertions can run. Keep the independent expected tree and the no-restoration checks.
2d2e9d0 (commit: retain suspended history across successful hooks, 2026-08-18) guards checkpoint capture with the required descriptor operations, but its recording and restoration paths still compile on Windows. Their direct calls to geteuid() break the native Windows build. Using Git for Windows' placeholder getuid() would not authenticate the owner of an index file. Keep the ownership predicate with the checkpoint code and enable it only where anchored file verification is available. Unsupported platforms decline the checkpoint. Check the hook's final owner through the same pinned descriptor when preparing the restore, so the index writer no longer needs its own POSIX-only call. The supported-platform ownership checks and historical-only proof semantics are unchanged.
Interactive add and scoped stash can refresh and rewrite the main index before enabling external fsmonitor history. During watch-limit backoff, even quitting an interactive prompt can then discard the pending token and untracked history. The apply and am entrypoints also reject every GIT_INDEX_FILE, including the canonical index passed by ordinary hooks. Centralize the pre-read setup around the existing physical-main-index check. Admit only the trusted repository namespace and an IPC provider or authenticated watch-limit backoff, then attach the repository config digest before reading the index. Reuse this setup in interactive add, scoped stash, apply, and am. Allow apply's existing history-preservation predicate to accept an explicitly selected canonical index, without loosening its patch-shape, attribute, or filter checks. This only prepares authenticated history reads; it does not grant current cleanliness or authorize temporary, alternate, or aliased index outputs. Cover scoped stash, genuine hook-invoked apply, canonical-index am, and the corresponding temporary-index rejection cases.
A successful commit -p writes its selected entries through a private index. That index cannot inherit proof authority from the main index, so the interactive child correctly discards fsmonitor history. Publishing the result then loses history even when the selection only changes the contents of existing regular files. Capture the parent's authenticated historical checkpoint before handing the temporary index to the interactive child. After selection, compare the held original descriptor with the parent's current index and the newly opened selected file. Advance the checkpoint only when entry names, modes, stages, and persistent flags agree, and every changed object passes the existing same-path attribute and filter checks. Recheck the original main index and repository configuration before sealing the selected entries. The ordinary successful-commit writer can then restore historical-only state at final publication. Subsequent hooks must leave the selected logical entries unchanged. The private index never gains authority, structural changes still revoke history, and every tracked entry remains dirty until a genuine token closure succeeds. Add quit and accepted-hunk interactive regressions, private-index and structural-selection controls, and a worktree-attribute recovery case. The attribute test permits historical retention but requires authoritative status and later recovery to detect the changed conversion rules.
Replacing a regular file while fsmonitor history is suspended poisons its cached stat data and sets CE_CONTENT_CHECK_REQUIRED. The zeroed stat tuple survives an index write, but that flag is intentionally memory-only. A later process can therefore mistake the poisoned tuple for an ordinary stat mismatch. In a scoped stash, add -u stages the correct worktree blob, then apply --index -R rejects that content-equal entry. Teach the shared stat/content-check wrapper to recognize a fully zeroed stat tuple in an authenticated suspended epoch. Use the existing ie_modified() path to verify content and refresh the stat data only after a match. Keep the gitlink bypass and ordinary unmarked zero-stat behavior unchanged. This recovers an obligation to check content, not authority to declare the worktree clean. The scoped-stash regression exercises the process boundary. Existing zero-stat and fsmonitor-content-recovery tests cover the ordinary and explicitly invalidated cases.
Interactive add exports the repository's selected index path as-is. In a primary worktree that can be .git/index rather than the absolute path used by the publication oracle. Derive the expected child spelling with rev-parse --git-path index, keeping the physical-index and private-lock assertions unchanged. The worktree-attributes fixture runs in a subshell, where test_hook cannot register its cleanup. Write the hook directly into the disposable repository instead. Both corrections pass against the unchanged candidate binary; neither changes the production behavior or proof assertions.
0d1e07f (fsmonitor: prepare history before canonical index reads, 2026-08-18) teaches scoped stash pushes to retain suspended fsmonitor history, but limits that setup to an ordinary push. A patch selection also refreshes the real index before prompting. During watch-limit backoff, even quitting the prompt can therefore discard FSMN and FSUC without changing the logical index. Stash apply and pop have the same problem when GIT_INDEX_FILE explicitly names the real main index. Their initial refresh rejects that spelling before any of the requested changes are applied. Use the shared physical-main initializer before either operation reads the index. Keep read admission independent of the eventual stash mode; the existing writers still validate their changes, and whole-worktree operations retain their conservative invalidation. Cover cancelled and accepted patch selections, canonical apply and pop, and private-index rejection.
217941b (dir: capture ignore sources beneath anchored parents, 2026-07-21) records a present but empty ignore file with a NULL buffer and zero length. The proof-capture interface accepts that representation but passes it to hash_object_file() after checking the source metadata. The block SHA-256 implementation then passes NULL to memcpy() while finishing the buffered blob header. UBSan rejects that call even though there are no content bytes to copy. Use the repository hash algorithm's empty-blob object ID when the captured size is zero. Keep the descriptor and namespace checks, the distinction between an empty file and a missing file, and the rejection of NULL with a nonzero size. Exercise those cases with both SHA-1 and SHA-256.
60077b9 (status: preserve semantic history across scoped and index changes, 2026-08-11) invalidates the current clean-status proof before saving a whole-worktree stash. That is too early for operations which may select nothing. Quitting an unscoped stash -p, or using --staged when only unstaged changes exist, drops the pending untracked history without changing the index or worktree. Keep the initial refresh authenticated, and defer this invalidation for patch and staged-only stashes until a nonempty selection succeeds. A successful patch selection leaves its private index in memory, so lock and reread the original selected index before invalidating and writing it. Real whole-worktree mutations retain their conservative invalidation before the stash is published or the worktree is changed. Cover both no-op paths, accepted selections with an independently staged sibling, and copied private indexes in primary and linked worktrees.
3d7c419 (status: preserve semantic history across scoped and index changes, 2026-08-11) taught unpack_trees() to transfer a current clean proof to a semantically equivalent index. During a watch-limit backoff, the index has only pending historical state, so that transfer rejects it. A clean, same-path stash merge then drops the untracked history even though its logical entries and subsequent recovery are correct. Capture the authenticated main-index source before unpacking and allow an explicit historical-only transfer after moving its extensions. Keep the original pending untracked cache only when the complete result has the same names, modes, stages, and persistent flags. Changed object IDs must also pass the existing attribute and filter checks. Any unsafe mutation permanently abandons the capture. The result remains all-dirty and cannot claim a current tracked or untracked proof. A checksum-disabled stash merge also replaces the source inode before its final index-only reset. Reread that committed index through the normal admission path instead of weakening the source-identity check. Cover canonical and implicit apply/pop, unsafe replacements, and successive zero-checksum publications with both object formats.
bf9f887 (unpack-trees: preserve suspended history across safe replacements, 2026-08-18) lets a clean stash merge retain suspended fsmonitor history. Its source must still identify the canonical index. With index.skipHash enabled, stash's initial refresh can replace that source inode. Restoring a distinct staged snapshot already rereads the index after reset_head(), but ordinary apply/pop and --index without a distinct snapshot skip that path. The merge then rejects the stale descriptor and drops pending untracked-cache history, even though the logical result is correct. Reread the suspended zero-checksum index before merging when the staged snapshot path has not already done so. Keep the existing post-merge reread and physical-source checks unchanged. Nonzero-checksum indexes, live fsmonitor operation, and the already-rebound staged-snapshot path do not incur another index read. Cover default apply/pop and the empty staged-snapshot path in both worktree layouts, including zero-trailer publication checks and copied private-index rejection.
3c98588 (status: reissue clean proofs after repository inputs change, 2026-08-13) lets ordinary status replace an otherwise valid sidecar when its repository fingerprint changes. A mismatch in the separate configuration digest returns earlier without requesting a replacement. A persistent nonsemantic change such as status.relativePaths=false can therefore leave a physically current but unusable sidecar in place. An ordinary status may validate the worktree and save resumable history without rewriting the index or issuing a new sidecar. Later invocations then repeat the index-reading fallback. Carry a configuration mismatch through the existing reissue signal. The old proof still fails validation. Issuance still requires an ordinary writable clean query and all existing proof checks. Do not treat the mismatch as a provider reset. Cover the transition with a zero-checksum index and scripted provider responses. Check that read-only status leaves both files alone, writable status changes only the sidecar, and the next identical command hits it without reading or writing the index.
5890aef (status: reissue clean proofs after config changes, 2026-08-18) lets ordinary status replace a sidecar whose configuration digest is stale. A temporary status.relativePaths override changes that digest too, so alternating ordinary status with "git -c status.relativePaths=false status" replaces the same sidecar on each transition. On a 1,160,465-entry checkout, the final ordinary status went from a 74 ms cache hit to a 341 ms index-reading fallback without changing the output or index. Exclude only command-scoped status.relativePaths from the proof digest. This is a presentation choice: fast status builds a fresh wt_status and prints with the current prefix instead of reusing cached output. Keep persistent configuration, missing scope metadata, other keys, and the legacy config-epoch admission unchanged. Flush pending filter settings before omitting the key so separated fragments cannot become a complete disabled-filter override. Cover the scope and filter boundaries with both hash algorithms. Retain the persistent-config repair test, require temporary A/B/A invocations to hit without index I/O or sidecar replacement, and compare dirty subdirectory output with independent status oracles.
dreynaud-oai
approved these changes
Aug 19, 2026
ttaylorr-oai
marked this pull request as ready for review
August 19, 2026 04:19
ttaylorr-oai
marked this pull request as draft
August 19, 2026 05:40
5890aef (status: reissue clean proofs after config changes, 2026-08-18) lets a writable status replace a proof after persistent configuration changes. A temporary color.ui or core.quotePath setting also changes the digest, so it replaces the default proof. The next default status then reads the index again and replaces that proof in turn. Before the reissue change, the temporary command missed but left the default proof usable. Extend the command-scoped presentation exemption from 44402f3 (status: ignore command-scoped relativePaths in clean proofs, 2026-08-18) to these two exact keys. A clean proof stores no rendered output. The ordinary configuration parser and current status printer still apply each setting. Keep persistent and unknown-scope entries in the digest, flush incomplete filter overrides before the exemption, and leave legacy tracked-policy admission unchanged. Cover writable default/override/default sequences without index I/O or sidecar replacement, and compare actual color and pathname quoting against the ordinary status path. Retain the persistent-config reissue and filter-boundary controls.
2c6c284 (status: retain clean proofs across presentation overrides, 2026-08-19) puts environment assignments directly before test_must_fail. That form is not portable for shell functions, so test-lint rejects the new invalid-configuration cases. Use test_env to export the settings in a subshell. Keep the expected parser failures, empty output, and unchanged index and sidecar checks.
dreynaud-oai
approved these changes
Aug 19, 2026
ttaylorr-oai
marked this pull request as ready for review
August 19, 2026 18:38
ttaylorr-oai
force-pushed
the
codex-unstable
branch
2 times, most recently
from
August 24, 2026 18:44
f906482 to
7e54125
Compare
ttaylorr-oai
added
kind:review-only
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current candidate:
4e82cbf14855This closes the temporary watch-limit backoff gap found in review. Clean
add --refresh, diff's optional stat refresh, and stash's optional canonical-index writes preserve the existing physical proof. A real staging write still publishes the requested contents, but retains only authenticated historical state: an all-dirty FSMN bitmap, an unbound FSCF manifest, and pending untracked candidates. Same-path replacements recheck root and ancestor attributes; unsafe mutations revoke that history. No clean proof can be issued until a fresh provider boundary and ordinary revalidation agree. Explicitcore.fsmonitor=falsebehavior is unchanged.The four new regressions fail for their intended reasons on immutable
f7be5b1d18under both hashes. The exact clean successor builds on macOS and passes 52 focused SHA-1/SHA-256 assertions, including all ten backoff tests, existing closure/recovery controls, sidecar identity attacks, and daemon compatibility. Unit results are 401 passed and one platform skip. Two later-successful-cookie checks correctly skip on the unhealthy native provider; they are not counted as successful native delivery coverage. The initial sandbox-denied native runs are retained, and the separately approved private-fixture rerun passes all mandatory cases. Both independent source reviews pass. Exact-head Linux normal, no-pthreads, LeakSanitizer, and ASan/UBSan qualification is next; fresh PR CI and push CI are running.Release hold remains: this Git-only writer fix does not repair the primary Mac's native cookie delivery or establish an FSEvents ordering guarantee. Keep PR59 draft and out of the live metadata plan. Exact-head human review, real-checkout correctness and warm performance, and explicit Mac promotion clearance remain required. No
og/notifypatch orog-previewpublication is part of this update. All source-specific results below are historical.Earlier candidate:
f7be5b1d18f9This removes only the
a8d82e3790asynchronous-flush backport. Upstream has retracted that topic, and our matched native experiment and large-worktree test did not establish a performance repair. The author's precise reason for withdrawal has not been verified. The resulting source tree is byte-identical to68abca224f29; all earlier correctness, daemon-compatibility, and bounded-retry fixes remain.The exact clean head builds on macOS and passes all unit tests and 34 focused SHA-1/SHA-256 assertions. Exact-head Linux qualification is complete: 74 normal suite arms (3,848 successes), 12 no-pthreads arms (218 successes; six native-only suites correctly skipped), 12 LeakSanitizer arms (470 successes), and 10 ASan/UBSan arms (249 successes). Each build also passes 392 unit tests with 10 skips. There are no unexpected failures. Only the no-pthreads build uses two scoped exemptions for inherited unused-parameter and unused-function warnings; all other developer warnings remain errors. Both PR CI and push CI passed 45 jobs with two expected skips; style and whitespace checks also pass. Independent exact-head review and the Mac release gates remain open.
Release hold remains: the primary large worktree still loses native cookie boundaries. Keep this PR draft and out of the live metadata plan. A private trace-only experiment confirms that some cookies arrive after the one-second deadline. A matched three-second-deadline experiment did not improve the timeout rate and is not included. Neither a longer wait nor a flush establishes the missing FSEvents ordering guarantee. No Mac release or
og-previewpublication is authorized. All source-specific results below are historical.Earlier candidate:
a8d82e379088This update backports upstream 08b12d90, which prompts Darwin FSEvents to deliver buffered notifications before the existing one-second cookie wait. The asynchronous call avoids blocking the listener on the daemon lock. The timeout and failed-cookie behavior are unchanged.
The preceding follow-ups make failed-cookie token retirement an explicit daemon compatibility requirement, bind that requirement to the actual query response, and preserve older client protocols without adding IPC to a marked warm query. Status also records a precise TRIVIAL result from its fast-sidecar probe and falls back after two consecutive closing resets, while preserving recovery from one transient reset.
The exact clean head builds on macOS and passes all unit tests and 34 focused SHA-1/SHA-256 assertions. The preceding
798fb21ad8passed all four Linux variants and the actual old/new/old client interoperability test;68abca224fpassed all 74 normal Linux suite arms. Exact-final-head Linux qualification and hosted CI are running. A matched 48-query native delivery experiment still observed 12 timeouts in each 24-query arm. A controlled switch of the primary large-worktree target to the exact patched server also failed its first closing query after one second; the guarded status was stopped before another full-tree scan. The backport is therefore not an accepted performance repair on that machine.Release hold remains: keep this PR draft and out of the live metadata plan. The asynchronous flush does not establish an FSEvents ordering guarantee. Actual-target performance, exact-head review, and the existing Mac promotion decision remain open. All source-specific results below are historical.
Earlier candidate:
bd208f35619fFour further commits repair the repeated-refresh path and the hosted
test failures. A status-owned write receipt now binds a rewritten
index.skipHashindex to the exact lockfile inode, records its identitybefore
post-index-change, and rechecks it after the hook. This permitssafe clean-sidecar republication without accepting a foreign replacement
or an in-place hook edit. A provider reset also schedules republication.
An exhausted token-closure retry no longer starts a full refresh that it
cannot close; the conservative fallback remains. Two test-only changes
fix shell-function environment export and a zero Windows permission bit.
The exact clean head builds on macOS and passes all unit tests plus 22
focused SHA-1/SHA-256 assertions, including same-inode and atomic-replace
hook attacks. All four Linux build variants pass; their complete normal,
reftable, no-pthreads, LeakSanitizer, and ASan/UBSan tests are running.
Fresh hosted CI and exact-head review are pending. The source-specific
results below are historical and do not replace these gates.
Release hold remains: replacing an old daemon with the current Git
did not restore cookie health on the primary performance target. The
bounded follow-up probe stopped before another full-tree scan. These
fixes do not claim to repair the separate Darwin provider-ordering issue
or establish acceptable real-checkout warm performance. Keep this PR
draft and out of the live plan until the existing promotion hold is
explicitly cleared.
Earlier correctness and recovery follow-up
The successor to
aaa01b4ec1adds focused fixes for failed providerboundaries, optional-index recovery, and avoidable metadata work:
and retire the daemon token after a failed synchronization cookie.
commitand realcheckout-indexupdates. Keep unchanged checkout operations no-write.rebuilding worktree metadata they do not need.
bound nested-attribute invalidation and reuse authenticated manifests
for wider directory deltas. Tolerate unrelated sibling churn only
around a held regular exclude source, retaining strict absent-source
and namespace checks.
recoverable misses. Keep corruption of the actual index fatal.
without reading past the path terminator.
speculative EOIE probes into high-bit payload bytes.
The earlier candidate was
be3eacfd884d229ea7d859e77f4624e5c69f29fa. Its normal Mac build, unittests, and controlled witness tests pass. Both new high-bit-signature
regressions reproduce the actual UBSan failure on the preserved previous
binary, and the corrected Linux ASan witness suite passes. The previous
head passed all 70 Linux normal-suite arms and all eight LeakSanitizer
arms. The complete exact-head Linux, sanitizer, compatibility, and hosted
CI matrix is still in progress. Results recorded below for
aaa01b4ec1and earlier commits remain historical; they do not qualify this head.
Hosted CI: pull request
and topic push.
Release hold: this PR is draft so the protected controller cannot
admit it while macOS promotion is held. A separate, pre-existing Darwin
provider-ordering issue can acknowledge a synchronization cookie before
an earlier worktree event is delivered. The provider-level failure is
confirmed; this update does not claim an end-to-end status reproduction
or a repair for that ordering issue. Promotion requires explicit Mac
clearance or a separately reviewed and authorized release-policy change.
Replacement-ref batching, additional index-offset-table optimizations,
and streaming-index work are deferred from this release candidate.
Problem
Ordinary index writers and provider resets can discard authenticated
clean-status history even when the tracked changes are safe. For example,
a cache-tree miss in
write-treecan retainFSMNandFSUCwhileweakening a fully bound
FSCFproof. Subsequent status, snapshot, andhook commands then rebuild the complete attribute manifest.
Recovery can also duplicate work: after discarding an unauthenticated
legacy untracked cache, read-only status can scan the worktree in bulk
and then walk the same directories again to collect untracked files.
An index writer must also preserve pending untracked-directory events:
serializing the advanced provider token without those invalidations can
make the next status omit a newly created untracked file.
Changes
Preserve authenticated configuration and worktree history through
physical
write-tree, ordinary index writers, reset, merge, rebase,replay, stash, and real primary-index lockfile hooks. Keep temporary
indexes, active filters, attribute changes, conflicts, and unsafe index
aliases on their existing fail-closed paths.
Let a genuine private copied index with partial manifest history use
the existing conservative stat fallback. Strongly invalidate tracked
and untracked hints without rebuilding a full worktree manifest; keep
canonical index, lockfile, inode-alias, attribute, and filter guards.
Give narrowly classified cached and scoped readers a conservative
bootstrap path. Invalidate hints instead of certifying incomplete
metadata, avoid the full manifest and stale-checkpoint digest, and
suppress optional index writes only when that fallback is used.
Repair provider-reset history before
diffpublishes its index. Reusepinned observations, keep the index lock unheld during traversal, and
recheck the physical index and all proof bindings before publishing.
Reuse a checked semantic manifest across authenticated, scoped directory
deltas. Bound reuse by the affected attribute sources, and fall back for
changed ignore rules, lost events, races, or incomplete proofs.
Share verified normalization of identical ignore-file contents, avoid an
unnecessary reopen of anchored regular sources, and admit clean
sidecars when configured filters are proven inactive. Preserve the same
correctness and progress behavior without pthreads.
Let whole-worktree, read-only status recover discarded populated or
empty legacy untracked caches using one complete provider-authenticated
bulk scan. Select bulk preload automatically only for this recovery;
explicit
core.preloadIndexBulk=falsestill opts out. Writable statuscontinues to rebuild durable authenticated untracked metadata.
Serialize unresolved fsmonitor directory events as invalid UNTR nodes.
Preserve paired history across authenticated same-path regular-file
replacements and scoped stash push/apply, without resurrecting dirty
cache entries. Mixed or unsafe patch batches remain conservative.
Initialize authenticated history before
amfirst reads or refreshesthe index. Preserve fully bound proofs at an unchanged provider token;
attribute-changing patches and temporary three-way indexes retain the
existing fail-closed behavior.
Normalize a command-scoped
submodule.recurse=falseonly when theeffective setting is already false. A guarded checkout can then leave
directly reusable linked-worktree proofs without making ordinary diffs
repeatedly hash an old checkpoint.
Fix the pre-existing one-byte SHA-256 stack overflow in
do_store_stash(). Giveoid_to_hex_r()room for its terminating NUL,and exercise an explicit SHA-256 repository in the ordinary stash suite
so the default SHA-1 sanitizer jobs cover the failing path.
Test the sidecar suite's native-provider prerequisite with a real
daemon query. A successful status may deliberately preserve the physical
index and publish external history instead, so the prerequisite must not
require a physical
FSMNextension.Give the external-exclude FIFO tests private parent directories.
Unrelated temporary-file churn must not invalidate their initial exclude
proof before the tests reach the intended race barrier. Keep the
production proof checks and all FIFO assertions unchanged.
Defer the full source-index digest for narrowly bounded Linux
status --untracked-files=nocalls with current, fully paired proofs.Keep normal content checks and the whole-index racy check. A real stat
repair reconstructs the original digest from the pinned reader FD and
follows the ordinary external-history and physical-index writer paths.
Zero-checksum indexes remain bound to a process-local physical epoch;
a changed source rolls back our lock without publishing or writing.
Release the temporary FSMN dirty bitmap after raw source-index
recapture. The raw reader intentionally skips normal fsmonitor
post-processing, so its shadow must free the bitmap before releasing
the index. Keep the proof and physical-write decisions unchanged.
These paths do not treat directory mtimes as proof that tracked-file
contents are unchanged.
Verification
Current hold:
aaa01b4ec19c0c269d0af240cdbe0a01d128b491fixesthe 56-byte raw-index repair leak found by both workflows' files and
reftable LeakSanitizer jobs on
cb6cbd4f5e. The temporary index owns anfsmonitor_dirtybitmap thatrelease_index()does not free. Thefailing job
identifies t7519 test 102 and the allocation stack. The fix frees and
clears that bitmap at the common cleanup label, following the existing
parsed-index cleanup pattern.
A matched Linux
SANITIZE=leakbuild reproduces the exact 40-bytedirect and 16-byte indirect leaks on unmodified
cb6cbd4f5e. With onlythe three-line cleanup, test 102 passes under SHA-1 and SHA-256 with both
files and reftable refs, with no sanitizer report files or deduplication
tokens. All 386 unit tests pass. An independent audit rechecks the raw
TAP, counts, exits, leak logs, build hashes, and each fixture's actual
object and ref formats.
The clean
aaa01b4ec1normal Linux build passes all 386 unit testsand both complete 102-case status suites. Four real-provider cases cover
primary and linked worktrees under both object formats. An independent
raw audit verifies 24 status traces: eight dirty calls skip the source
digest and physical write, four clean repairs restore the original
source, and four foreign-writer followers recover read-only. All owned
fixture daemons were stopped.
The clean Mac build passes 386 unit tests,
check-meson,test-lint,nine focused status regressions per object format, two explicit-SHA-256
stash controls, and 12 real-provider sidecar assertions. The no-pthreads
build passes 386 units and 206 shell assertions; ASan/UBSan passes
386 units and all 86 selected assertions. The complete native matrix
passes 725 assertions with 11 expected skips. The full normal runtime
was restored and independently checked for sanitizer contamination.
Four write-denied comparisons on the actual checkout agree with a fresh
strong-stat oracle and leave its index, sidecars, and daemon unchanged.
Those calls missed the clean cache, so they are not a warm-hit benchmark.
The compatibility matrix passes all 154 suite/hash runs: 4,390
successes, 36 expected skips, 212 existing expected failures, and no
unexpected failures. Four final 2,050-entry real-daemon cases cover
primary and linked worktrees under both hashes, with eight verified
recovery writes and eight immutable read-only followers. The strict
ten-part gate and independent final attestation pass. Local qualification
is complete; exact-head review and protected release integration remain
pending.
The preceding clean
cb6cbd4f5ebuild also passed the ordinaryexplicit-SHA-256 stash regression under both defaults and six
real-provider sidecar cases per hash. Its no-pthreads build passed
386 units and 206 shell assertions; its ASan/UBSan build passed
386 units and all 86 selected assertions; its complete native matrix
passed 725 tests with 11 expected skips. Those results remain tied to
cb6cbd4f5e; they do not clear its hosted Linux leak.The six files added in
cb6cbd4f5ematch the independently reviewed andqualified Linux candidate byte for byte. The new Linux-only regression
fails for the intended missing optimization on immutable
e94acd7cc2and passes on the candidate under SHA-1 and SHA-256. It covers primary
and linked worktrees, a real
index.skipHashzero-checksum index, agenuinely dirty zero-stat entry, clean stat repair with external-history
publication, selected and unselected racy entries, active and required
filters, and a deterministic zero-checksum foreign-writer race.
That Linux source also passes four real-daemon primary/linked worktree
and hash-format cases, plus 19 full regression suites under both hashes:
1,713 successes, 121 prerequisite skips, eight existing expected
failures, and no unexpected failures. A separately audited large-index,
scripted-provider comparison confirms that the selected dirty command
avoids the full logical digest while preserving the independent status
oracle, physical index, and checkpoints. Real-daemon correctness and
synthetic scale performance are separate results. The index read and
whole-index racy check remain; this is not a constant-time status path.
Earlier source-specific regressions and matched controls
e94acd7cc2c7087a6e097e2b87251b04c374e59dwas the precedingreview candidate. Its clean build passes 386 unit tests,
check-mesonand
test-lint, and nine focused status, copied-index, race, dirty-UNTR,linked-precommit, mail-application, and native linked-checkpoint
regressions under both SHA-1 and SHA-256. The new ordinary stash test also
passes under both default hash modes; it creates a SHA-256 repository in
either mode. The corrected native sidecar prerequisite and six selected
sidecar cases, including both external-exclude FIFO tests, also pass under
both hashes, with actual provider-query evidence. The preceding
284a8ae410head also passed all nine scoped-stashcases and the bounded-reader matrix in both object formats with the
complete
linux-TEST-varsenvironment.The exact stash correction has paired ASan/UBSan evidence. An immutable
fec9824447build passes ordinary SHA-1 stash operations, but the newexplicit SHA-256 test reproduces the stack-buffer-overflow under both
default hash modes. With only the one-byte buffer correction and that
regression, the precommit build passes 386 units and the complete ordinary
and scoped stash suites under both hashes: 308 successes, four existing
known breakages, two expected skips, and no unexpected failures. This
precommit result is not a substitute for the clean successor's final
sanitizer qualification.
The clean
687de436f1no-pthreads gate passed 386 units and 206assertions. Its sanitizer build passed 386 units, then the selected
sidecar suite exposed an inherited prerequisite false negative. Matched
native controls on immutable predecessor and candidate builds reproduce
the failure under both hashes: non-racy status stores external history,
leaves the physical index unchanged, and the old prerequisite incorrectly
reports no fsmonitor. Checking the real daemon response passes all four
controls while preserving that no-write behavior. The corrected
prerequisite also passes all eight selected sidecar sanitizer cases.
ef3af6c978changes only this test setup; production is unchanged from687de436f1.The clean
ef3af6c978no-pthreads gate passes 386 units and 206assertions. Its ASan/UBSan build also passes 386 units and all 82 selected
assertions under both hashes, including the ordinary explicit-SHA-256
stash regression and all eight native sidecar cases. Its hosted sanitizer
job also passes the full ordinary stash suite. These results remain tied
to
ef3af6c978.The clean
e94acd7cc2no-pthreads gate passes 386 unit tests and 206assertions. Its ASan/UBSan build passes 386 units and all 86 selected
assertions across 12 suite/hash runs. Both default hashes execute the
explicit SHA-256 stash regression and all six native sidecar cases,
including both FIFO tests. The authoritative counts, exact sanitizer
binary and helper hashes, raw logs, and build provenance are retained.
That head subsequently passed all 12 native suite/hash runs (725
successes and nine expected skips) and all 154 compatibility runs
(4,390 successes). These remain e94 results.
The external-exclude fixture correction has a matched causal control.
Unrelated sibling churn reproduces the exact pre-barrier
fast-excludesfallback on immutable parent and candidate builds under both hashes. The
exclude file itself remains unchanged. All four controls pass when the
fixture uses its own parent directory, and still perform the real FIFO
replacement and require
fast-excludes-raced. The original quiet controlalso passes. The proposed test-only change then passes all 12 selected
native sidecar assertions under the pinned
ef3af6c978sanitizer build.The specific parent mutation in the hosted failure was not recorded; the
controlled comparison establishes the mechanism without weakening it.
The new
amfix passed 100 affected suite/hash runs before commit:1,296 successes, ten existing known breakages, and no unexpected failures.
The same-token positive and changed-attributes negative pass under both
object formats. The clean
810ab59f8dcommit also passed four real-daemonprimary/linked SHA-1/SHA-256 cases: immediate fully paired physical proofs,
correct commit and worktree contents, and eight immutable read-only
followers with no manifest scans or index writes. The subsequent
fec9824447commit changes only test setup;687de436f1adds the separatestash buffer correction described above.
ef3af6c978corrects thesidecar prerequisite, and
e94acd7cc2isolates the two FIFO fixtures.The test-only follow-ups register the new suites with Meson, correct
negative grep assertions, and give the same-stat and required-filter
controls an unchanged-staged-content scratch index that forces content
inspection. The staged-sibling fixture also clears inherited split-index
mode in the subshell that actually runs the scenario. Its original
failure and the one-line correction were reproduced under both object
formats; the full-proof assertions remain unchanged. The linked-history
fixture also ages its newly staged file before adding it. A native
future-mtime control reproduces the original
racy-indexcheckpointrejection on current and parent-equivalent builds in both hashes; the
corrected fixture preserves every checkpoint and proof assertion.
The exact
284a8ae410source also passed all 154 compatibility suite/hashruns: 4,388 successes, 36 skips, and 212 existing known breakages, with
no unexpected failures. Its complete native matrix passed all 12
suite/hash runs: 721 successes, nine expected skips, and no failures.
An earlier restricted run stopped before establishing its initial
provider-bound proof. The unchanged regression passes on both immutable
baseline and candidate builds, under both object formats, with normal
platform permissions; that earlier restricted-run trigger remains unknown.
The exact
fec9824447head subsequently passed all 154 compatibilitysuite/hash runs (4,388 successes), all 12 native runs (725 successes), and
the no-pthreads gate (204 successes). Its sanitizer gate then exposed the
pre-existing SHA-256 stash overflow, so it is not release-qualified.
Those results remain specific to their recorded source heads.
The scoped-stash source at
428d829240passed 1,832 assertions across120 suite/hash runs, including all nine new independent-oracle stash and
indexed-apply cases, the apply and stash compatibility suites, and
required-filter, mixed-patch, alternate-index, and pending-untracked-event
controls. The subsequent config-normalization source passed the same
nine stash cases, the bounded-reader matrix, and six focused status
regressions in both object formats.
The earlier complete recovery qualification remains specific to
3fcb4c5a12b6c699f1a85cc3f5601d6c6578d530:skips, no failures.
breakages, no unexpected failures.
coverage: 50 successes.
object formats and primary/linked worktrees, with independent
offline-change oracles.
read-only recovery, identical status output, and unchanged physical
index, sidecars, and daemon.
Review and release scope
The
cb6cbd4f5ePR CI runand push CI run
reproduced the LeakSanitizer failure described above. Its style and
whitespace checks passed. The corrected
aaa01b4ec1PR CI run and
push CI run
each passed 45 jobs with two expected skips. All four files/reftable
leak-sanitizer jobs, ASan/UBSan, style, and whitespace checks pass.
The only initial PR failure was Debian's package-mirror download reset
before checkout. The same-head push Debian job passed, and one
infrastructure-only retry
passed the complete job. The original failed log remains retained.
Independent approvals for
dfc55f903ca1ande94acd7cc2remain tied tothose commits; fresh approval and platform CI must cover
aaa01b4ec1.Release qualification still includes temporary-index commands, scoped
readers, stash push/apply, coherent packaged writers, and linked
worktrees on macOS and Linux. The release remains held until the
exact-head tests, independent review, and protected controller's
reviewed integration are complete.
The new Linux optimization does not admit other platforms, broad or
implicit pathspecs, unsafe filters, incomplete proofs, or changed
physical index epochs. Those cases retain the existing conservative
path. The separate experimental writable directory-stat reuse change
was dropped after its controlled performance comparison.
Earlier hosted CI and fixture investigations
Hosted CI on
116bc02570found missing Meson registrations andtest-greplint errors. The next head reached substantive tests and exposed
the stale-stat required-filter control and inherited split-index fixture.
The corrected
284a8ae410head passed all 92 executed hosted checks,with seven intentional skips. Both complete CI workflows passed, including
all previously failing configurations, leaks, and sanitizers. The next
810ab59f8dhead added the separately testedaminitialization fix.Its complete PR workflow passed; the parallel push workflow exposed the
pre-existing linked-checkpoint timestamp race described above.
fec9824447fixes that fixture; its completePR CI run and
push CI run each
passed 45 jobs with two expected skips. Those workflows did not combine
SHA-256 with sanitizers. The
687de436f1head adds the buffer correctionand an explicit SHA-256 regression that closes that gap. Its complete
push CI run
passed 45 jobs with two expected skips; the sanitizer job executed the
new ordinary stash test. The test-only
ef3af6c978prerequisite fix thenpassed its complete PR CI run.
Its parallel push CI run
passed 44 jobs and failed only the existing raced-exclude-FIFO fixture on
macOS GCC, with two expected skips. The same exact head and compiler
passed in the PR workflow; both Meson jobs passed all 50 sidecar tests.
The retained failure is a successful conservative
fast-excludesfallback before the intended barrier, not a blocked FIFO read. The
matched parent-churn controls above reproduce that failure and validate
the fixture-only correction in
e94acd7cc2. ItsPR CI run and
push CI run
each passed 45 jobs with two expected skips. The PR workflow's initial
macOS Clang run failed an existing fsmonitor event assertion; one
same-head rerun passed the full job. The original failure remains
retained, and its precise trigger is unproven. No production check was
weakened.
The preceding
b208bc4f2298bdd37279f3b8e8835a07b82254fbhead passedall 92 hosted checks, with seven intentional skips. That includes both
workflows' AlmaLinux, linux32, and Linux test-variable jobs, which confirm
the optional-lock isolation and successful-cleanup-trap fixes.