Skip to content

Split blob-hydration and directory-enumeration failure telemetry by cause - #2071

Merged
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/split-hydration-enum-telemetry
Aug 5, 2026
Merged

Split blob-hydration and directory-enumeration failure telemetry by cause#2071
tyrielv merged 1 commit into
microsoft:masterfrom
tyrielv:tyrielv/split-hydration-enum-telemetry

Conversation

@tyrielv

@tyrielv tyrielv commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

The GVFS release-readiness telemetry has two error buckets that mix causes outside gvfs.exe's control (network, local disk/IO, ProjFS) with actionable ones (a missing object on the server, a size mismatch, or GVFS's own stale-enumeration eviction), so a release manager can't tell whether a spike is fixable:

  • Blob hydration failure — TryCopyBlobContentStream can't serve file contents to ProjFS
  • Directory enumeration failure — Failed to find active enumeration ID

This stamps a cause tag on the failure telemetry so the dashboard can split each bucket. No behavior changes — metadata only.

Changes

  • BlobHydrationFailureCategory (new enum) on the terminal blob-hydration errors:
    • Not gvfs-fixable: NetworkUnavailable, LocalIO, ProjFSWriteFailed
    • Actionable: ObjectNotOnServer, LocalCopyFailed, SizeMismatch, Unexpected
    • The root cause was previously collapsed into GVFSGitObjects.TryCopyBlobContentStream's bool return before the error was logged; it's now captured. The size-mismatch / IOException / WriteFileData failure sites were logged with a message the dashboard didn't match — they now carry the tag so they're counted.
  • EnumerationFailureReason on Failed to find active enumeration ID:
    • Evicted — GVFS's stale-enumeration eviction (gvfs.max-active-enumerations, off by default) removed a live enumeration (self-inflicted).
    • Unknown — ProjFS delivered an id GVFS never held or already ended.

Testing

  • Full unit suite: 888 passed, 0 failed (11 pre-existing skips).
  • New/extended tests assert the tags: TerminalBlobHydrationFailureIsTaggedWithCategory, GetDirectoryEnumerationTagsEvictedVersusUnknownId, OnGetFileStreamHandlesWriteFailure.

Target branch

master — observability enrichment for the shippable line, no behavior change; the enumeration-eviction feature this diagnoses already lives on master.

@tyrielv
Split blob-hydration and directory-enumeration failure telemetry by c…
46c694f 
…ause

The GVFS telemetry "Blob hydration failure" and "Directory enumeration
failure" buckets conflate causes outside gvfs.exe's control (network,
local disk/IO, ProjFS) with actionable ones (a missing object on the
server, a size mismatch, or GVFS's own stale-enumeration eviction).

Stamp a cause tag on the failure telemetry so the release-readiness
dashboard can bucket them apart. No behavior changes:

- BlobHydrationFailureCategory (nested in GVFSGitObjects) is now returned
  from TryCopyBlobContentStream via an out parameter as well as stamped on
  the terminal telemetry, so the virtualizer's own terminal event is tagged
  with the same cause rather than left uncategorized. Categories:
  NetworkUnavailable / DownloadFailed / LocalIO / ProjFSWriteFailed (not
  gvfs-fixable) vs ObjectNotOnServer / LocalCopyFailed / SizeMismatch /
  Unexpected (actionable). The size-mismatch, IOException, and WriteFileData
  failure sites were previously logged with a message the dashboard did not
  match; they now carry the tag so they are counted.

- EnumerationFailureReason (nested enum) on "Failed to find active
  enumeration ID": Evicted (GVFS eviction removed a live enumeration;
  self-inflicted) vs Unknown (ProjFS delivered an id GVFS never held or
  already ended). The eviction-tracking map is populated before the entry
  is removed from the active set (closing a mislabel race) and pruned on
  every sweep so it cannot outlive its window.

Unit tests assert each cause value deterministically.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv force-pushed the tyrielv/split-hydration-enum-telemetry branch from a667a05 to 46c694f Compare July 20, 2026 19:12
@tyrielv
tyrielv marked this pull request as ready for review August 5, 2026 18:22
This was referenced Aug 5, 2026
Open
Closed
Open
Merged
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 5, 2026
@tyrielv
Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA
7d4d7ba 
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 5, 2026
@tyrielv
Attribute wrapped blob-hydration failures to their inner cause
c788f02 
RetryableException wraps its real cause in InnerException. The blob-
hydration failure categorization checked the RetryableException type
itself, so every RetryableException - the largest hydration failure
bucket in the field - was tagged NetworkUnavailable, even when the real
cause was local disk/IO.

On this branch the RetryableException reaches OnFailure from
Context.Repository.TryCopyBlobContentStream - typically StreamUtil
wrapping an IOException while reading a corrupt or truncated local loose
object. Unwrap RetryableException.InnerException before categorizing, and
map IOException / UnauthorizedAccessException / Win32Exception (the local
disk/IO family) to LocalIO. A RetryableException whose inner cause is not
local (e.g. HttpRequestException), or that has no inner cause, stays
NetworkUnavailable. Telemetry metadata only; no behavior change.

Add unit tests for each inner-cause mapping (IOException, Unauthorized-
AccessException, Win32Exception -> LocalIO; HttpRequestException and no
inner -> NetworkUnavailable), and reset the process-global
RetryCircuitBreaker in the fixture SetUp and TearDown so these failure-
driving tests cannot open the circuit for one another or for a later
fixture.

Stacked follow-up to PR microsoft#2071; do not publish until microsoft#2071 merges.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 5, 2026
@tyrielv
Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA
902a99b 
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.
- WindowsFileSystemVirtualizer routes the request's logged sha through
  ToLoggableShaString, so a malformed content-id can no longer enter
  telemetry with raw NUL/control bytes at the terminal hydration-failure
  error either (a no-op for a valid hex SHA).

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
tyrielv merged commit 0fc73a2 into microsoft:master Aug 5, 2026
35 checks passed
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 5, 2026
@tyrielv
Attribute wrapped blob-hydration failures to their inner cause
13794ab 
RetryableException wraps its real cause in InnerException. The blob-
hydration failure categorization checked the RetryableException type
itself, so every RetryableException - the largest hydration failure
bucket in the field - was tagged NetworkUnavailable, even when the real
cause was local disk/IO.

On this branch the RetryableException reaches OnFailure from
Context.Repository.TryCopyBlobContentStream - typically StreamUtil
wrapping an IOException while reading a corrupt or truncated local loose
object. Unwrap RetryableException.InnerException before categorizing, and
map IOException / UnauthorizedAccessException / Win32Exception (the local
disk/IO family) to LocalIO. A RetryableException whose inner cause is not
local (e.g. HttpRequestException), or that has no inner cause, stays
NetworkUnavailable. Telemetry metadata only; no behavior change.

Add unit tests for each inner-cause mapping (IOException, Unauthorized-
AccessException, Win32Exception -> LocalIO; HttpRequestException and no
inner -> NetworkUnavailable), and reset the process-global
RetryCircuitBreaker in the fixture SetUp and TearDown so these failure-
driving tests cannot open the circuit for one another or for a later
fixture.

Stacked follow-up to PR microsoft#2071; do not publish until microsoft#2071 merges.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 7, 2026
@tyrielv
Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA
882b7d8 
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.
- WindowsFileSystemVirtualizer routes the request's logged sha through
  ToLoggableShaString, so a malformed content-id can no longer enter
  telemetry with raw NUL/control bytes at the terminal hydration-failure
  error either (a no-op for a valid hex SHA).

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 7, 2026
@tyrielv
Fail blob hydration cleanly on a malformed (NUL-byte) placeholder SHA
2d88803 
When a user process reads a virtualized placeholder whose stored
content-id is corrupt - specifically 40 NUL bytes instead of a hex SHA -
GVFS builds a loose-object path from it and Path.Combine throws
System.ArgumentException ("Illegal characters in path").

ArgumentException is not in RetryWrapper.IsHandlableException, so it
bypasses both the retry logic and the download fallback in
GVFSGitObjects.TryCopyBlobContentStream and propagates to the
virtualizer's outer catch, which returns FileNotAvailable to ProjFS. The
placeholder can never hydrate, so the failing read repeats forever - a
retry storm. This is the #1 blob-hydration failure cause on the LKG field
build 1.0.26014.1 (38 machines; ~61 machines / ~6.2K events across 30d;
one machine emitted ~2.49M error events).

This is a corrupt content-id, NOT GVFSConstants.AllZeroSha: AllZeroSha is
40 ASCII '0' characters, which yields directory "00" and does not throw.

Reject a malformed SHA before it is turned into a path:

- GitRepo.GetLooseBlobState returns LooseBlobState.Invalid (a clean,
  non-retryable miss) for a SHA that is not 40 hex characters, so
  Path.Combine can never throw here again.
- GitRepo.LooseObjectExists guards the same Path.Combine.
- GVFSGitObjects.TryCopyBlobContentStream short-circuits a malformed SHA
  before the retry loop, so a bogus SHA never triggers a doomed 404
  download or a retry storm.
- SHA1Util.IsValidShaFormat is now null-safe; SHA1Util.ToLoggableShaString
  renders the bad value with non-hex characters escaped so telemetry stays
  greppable and free of control characters.
- WindowsFileSystemVirtualizer routes the request's logged sha through
  ToLoggableShaString, so a malformed content-id can no longer enter
  telemetry with raw NUL/control bytes at the terminal hydration-failure
  error either (a no-op for a valid hex SHA).

All three guard sites emit the same greppable Warning event
(*_MalformedBlobSha) at Warning level with no unhandled exception. Per an
existing decision this case stays telemetry category "Unexpected"; no new
BlobHydrationFailureCategory is added.

Stacked on microsoft#2071 (tyrielv/split-hydration-enum-telemetry): this branch is
rebased onto it, so microsoft#2071's out BlobHydrationFailureCategory parameter is
honored - the malformed-SHA short-circuit sets failureCategory =
Unexpected, so the virtualizer's terminal telemetry tags the case exactly
as before (it no longer reaches the outer catch because it no longer
throws). This PR must NOT merge before microsoft#2071; after microsoft#2071 lands, rebase
onto master.

Unit tests assert that a 40-NUL-byte SHA, a 40-char SHA with an embedded
path-illegal character, and other malformed SHAs return false from both
GitRepo.TryCopyBlobContentStream and GVFSGitObjects.TryCopyBlobContentStream
with no ArgumentException (Assert.DoesNotThrow), that no download/retry is
attempted, and that the out category is Unexpected.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
This was referenced Aug 10, 2026
Merged
Draft
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 10, 2026
@tyrielv
Repair a corrupt (NUL-byte) placeholder SHA at read time
49b77d1 
When a user process reads a virtualized placeholder whose stored content-id
is corrupt - 40 NUL bytes instead of a hex blob SHA - GVFS cannot hydrate it
from the corrupt content-id. microsoft#2074 makes that read fail cleanly (no crash, no
retry storm). This change goes one step further and repairs the underlying
data so the file works again.

Telemetry shows this corruption is durable and localized: the same 1-4 files
per machine fail repeatedly over multiple days until something rewrites the
placeholder (~61 machines / ~6.2K events over 30 days). It is old and
version-agnostic (spans >=4 GVFS builds), not a 2.0 regression. The triggering
processes are readers (git.exe, copilot.exe, Code.exe); the placeholder was
already corrupt on disk. The authoritative path->SHA still exists, because the
path is still projected and the git index projection can return the correct
SHA for it.

Read-time self-heal (WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler):

- Plumb virtualPath into the handler and, when the placeholder's decoded SHA is
  not a valid hex SHA, recover the authoritative SHA for virtualPath from
  GitIndexProjection.GetProjectedFileInfo and hydrate the blob from that instead
  of the corrupt content-id.
- The recovery passes a null BlobSizesConnection: repair needs only the SHA, not
  the blob size, so size resolution (which can throw SizesUnavailableException)
  is skipped and a size-lookup fault cannot deny a SHA-only self-heal.
- A successful hydration writes the whole file, which converts the placeholder
  into a full file on disk. The corrupt content-id is superseded and future
  reads never call back, so the file is repaired for good.
- If the path is no longer projected (deleted/renamed), the projection lookup
  throws, or the recovered SHA cannot be hydrated, fall back to the same clean,
  non-crashing FileNotAvailable failure as microsoft#2074.

We deliberately do NOT rewrite the placeholder's content-id in place via
UpdateFileIfNeeded. Confirmed empirically against real inbox ProjFS with a
throwaway probe: (1) serving the full content converts the placeholder to a
full file, so a second read issues no GetFileData callback (hydration alone is
the repair); and (2) UpdateFileIfNeeded on the file mid-read returns
0x80070020 (ERROR_SHARING_VIOLATION) because the reader holds the file open.
The same probe showed a corrupt placeholder is only injectable from the owning
virtualization instance (WritePlaceholderInfo accepts an all-NUL content-id)
and that an external FSCTL_SET_REPARSE_POINT rewrite is blocked (ERROR 1359),
so this behavior is covered by unit tests rather than a functional test.

Telemetry funnel (paired with microsoft#2074's *_MalformedBlobSha detection):

- Repaired: *_MalformedBlobShaRepaired (Warning) with the recovered SHA, so we
  can watch the corrupt-placeholder population drain.
- Repair miss: *_MalformedBlobShaRepairFailed (Warning) tagged with a
  MalformedShaRepairFailureReason (ProjectionMiss / ProjectionException /
  HydrateFailed / HydrateException). The failed event is emitted on every
  repair-failure exit - including hydration failures that throw after a SHA is
  recovered (size mismatch, local IO, ProjFS write failure) - so that
  repaired + repair-failed accounts for every repair attempt.

Coordinates with microsoft#2071: a repair miss stays telemetry category Unexpected; a
successful repair simply succeeds. No new BlobHydrationFailureCategory value.

Two known, accepted behaviors are documented in code: the projection is read
live, so a concurrent checkout can change the projected SHA between placeholder
open and repair (serving the currently-projected SHA is the best answer for an
already-corrupt file and matches the placeholder-creation path); and an
unrepairable-but-projected file whose blob is unavailable pays the normal
download + retry budget per read (the same cost any valid-but-unavailable
placeholder pays), bounded and never re-crashing.

Stacked on microsoft#2074 (tyrielv/fix-invalid-sha-hydration), which is stacked on
microsoft#2071. Targets vnext: this is a new behavioral change on the read path for an
old, rare, pre-existing corruption, so it does not belong on the 2.0
stabilization line. microsoft#2074 already removes the crash and retry storm on master.

Unit tests (WindowsFileSystemVirtualizerTests) cover: repair success (asserting
hydration uses the RECOVERED SHA, not the corrupt one) emits
*_MalformedBlobShaRepaired and completes Ok; a non-projected path, a throwing
projection lookup, an unhydratable recovered SHA, and a hydration that throws
after recovery each emit *_MalformedBlobShaRepairFailed with the expected
reason and fail cleanly; mid-repair cancellation emits neither repair event;
and a valid content-id still hydrates with no repair telemetry.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv tyrielv mentioned this pull request Aug 11, 2026
Merged
tyrielv added a commit to tyrielv/VFSForGit that referenced this pull request Aug 13, 2026
@tyrielv
Repair a corrupt (NUL-byte) placeholder SHA at read time
1fa041e 
When a user process reads a virtualized placeholder whose stored content-id
is corrupt - 40 NUL bytes instead of a hex blob SHA - GVFS cannot hydrate it
from the corrupt content-id. microsoft#2074 makes that read fail cleanly (no crash, no
retry storm). This change goes one step further and repairs the underlying
data so the file works again.

Telemetry shows this corruption is durable and localized: the same 1-4 files
per machine fail repeatedly over multiple days until something rewrites the
placeholder (~61 machines / ~6.2K events over 30 days). It is old and
version-agnostic (spans >=4 GVFS builds), not a 2.0 regression. The triggering
processes are readers (git.exe, copilot.exe, Code.exe); the placeholder was
already corrupt on disk. The authoritative path->SHA still exists, because the
path is still projected and the git index projection can return the correct
SHA for it.

Read-time self-heal (WindowsFileSystemVirtualizer.GetFileStreamHandlerAsyncHandler):

- Plumb virtualPath into the handler and, when the placeholder's decoded SHA is
  not a valid hex SHA, recover the authoritative SHA for virtualPath from
  GitIndexProjection.GetProjectedFileInfo and hydrate the blob from that instead
  of the corrupt content-id.
- The recovery passes a null BlobSizesConnection: repair needs only the SHA, not
  the blob size, so size resolution (which can throw SizesUnavailableException)
  is skipped and a size-lookup fault cannot deny a SHA-only self-heal.
- A successful hydration writes the whole file, which converts the placeholder
  into a full file on disk. The corrupt content-id is superseded and future
  reads never call back, so the file is repaired for good.
- If the path is no longer projected (deleted/renamed), the projection lookup
  throws, or the recovered SHA cannot be hydrated, fall back to the same clean,
  non-crashing FileNotAvailable failure as microsoft#2074.

We deliberately do NOT rewrite the placeholder's content-id in place via
UpdateFileIfNeeded. Confirmed empirically against real inbox ProjFS with a
throwaway probe: (1) serving the full content converts the placeholder to a
full file, so a second read issues no GetFileData callback (hydration alone is
the repair); and (2) UpdateFileIfNeeded on the file mid-read returns
0x80070020 (ERROR_SHARING_VIOLATION) because the reader holds the file open.
The same probe showed a corrupt placeholder is only injectable from the owning
virtualization instance (WritePlaceholderInfo accepts an all-NUL content-id)
and that an external FSCTL_SET_REPARSE_POINT rewrite is blocked (ERROR 1359),
so this behavior is covered by unit tests rather than a functional test.

Telemetry funnel (paired with microsoft#2074's *_MalformedBlobSha detection):

- Repaired: *_MalformedBlobShaRepaired (Warning) with the recovered SHA, so we
  can watch the corrupt-placeholder population drain.
- Repair miss: *_MalformedBlobShaRepairFailed (Warning) tagged with a
  MalformedShaRepairFailureReason (ProjectionMiss / ProjectionException /
  HydrateFailed / HydrateException). The failed event is emitted on every
  repair-failure exit - including hydration failures that throw after a SHA is
  recovered (size mismatch, local IO, ProjFS write failure) - so that
  repaired + repair-failed accounts for every repair attempt.

Coordinates with microsoft#2071: a repair miss stays telemetry category Unexpected; a
successful repair simply succeeds. No new BlobHydrationFailureCategory value.

Two known, accepted behaviors are documented in code: the projection is read
live, so a concurrent checkout can change the projected SHA between placeholder
open and repair (serving the currently-projected SHA is the best answer for an
already-corrupt file and matches the placeholder-creation path); and an
unrepairable-but-projected file whose blob is unavailable pays the normal
download + retry budget per read (the same cost any valid-but-unavailable
placeholder pays), bounded and never re-crashing.

Stacked on microsoft#2074 (tyrielv/fix-invalid-sha-hydration), which is stacked on
microsoft#2071. Targets vnext: this is a new behavioral change on the read path for an
old, rare, pre-existing corruption, so it does not belong on the 2.0
stabilization line. microsoft#2074 already removes the crash and retry storm on master.

Unit tests (WindowsFileSystemVirtualizerTests) cover: repair success (asserting
hydration uses the RECOVERED SHA, not the corrupt one) emits
*_MalformedBlobShaRepaired and completes Ok; a non-projected path, a throwing
projection lookup, an unhydratable recovered SHA, and a hydration that throws
after recovery each emit *_MalformedBlobShaRepairFailed with the expected
reason and fail cleanly; mid-repair cancellation emits neither repair event;
and a valid content-id still hydrates with no repair telemetry.

Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@KeithIsSleeping Keith Klein (KeithIsSleeping) KeithIsSleeping approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tyrielv @KeithIsSleeping