Skip to content

Split enumeration-miss cause (EndedRecently vs NeverSeen) + de-duplicate error telemetry - #9

Closed
tyrielv wants to merge 1 commit into
tyrielv/split-hydration-enum-telemetryfrom
tyrielv/enum-miss-split
Closed

Split enumeration-miss cause (EndedRecently vs NeverSeen) + de-duplicate error telemetry#9
tyrielv wants to merge 1 commit into
tyrielv/split-hydration-enum-telemetryfrom
tyrielv/enum-miss-split

Conversation

@tyrielv

@tyrielv tyrielv commented Aug 5, 2026

Copy link
Copy Markdown
Owner

What

Stacked on microsoft#2071 (base branch tyrielv/split-hydration-enum-telemetry). This extends the
directory-enumeration cause tagging with a finer split and removes a telemetry noise source. No
behavior change
- metadata and log-volume only; the returned HResult is unchanged.

Why

On the current LKG (1.0.26014.1), "Directory enumeration failure" ("Failed to find active
enumeration ID") is the largest named regression signature: ~9.7% of os.2020 mount machines hit it
in 30d. microsoft#2071 tags the miss as Evicted vs Unknown. Field analysis of that Unknown bucket shows
two distinct causes that need to be told apart, plus a per-machine event storm that inflates the
signal:

  • 100% of misses have restartScan == false; 96% occur >60 min after mount (steady state, not a
    startup/projection-not-ready race).
  • The kernel (ProjFS / gvflt) can deliver a GetDirectoryEnumeration that races the
    EndDirectoryEnumeration for the same handle - a query in flight while the directory handle is
    closing, or the querying process dying mid-enumeration. This is a benign close/query race, not an
    ID GVFS never held.
  • A caller re-enumerating a handle whose Start GVFS lost (for example across a provider restart) can
    re-issue the same failing Get in a tight loop; a single stuck enumeration has produced millions of
    error events on one machine, drowning the machine-based signal.

Changes

  • Split the miss reason. EnumerationFailureReason renames Unknown -> NeverSeen and adds
    EndedRecently:

    • Evicted - GVFS's own stale-enumeration eviction removed a live enumeration (self-inflicted).
    • EndedRecently - a Get raced or followed the End for the same enumeration (benign kernel
      close/query race; outside gvfs.exe's control).
    • NeverSeen - GVFS never held the ID: never started, or from before a provider restart (outside
      gvfs.exe's control).
      EndDirectoryEnumerationCallback records the ended ID in a recentlyEndedEnumerations map
      before removing it from activeEnumerations, so a racing Get always finds the ID in one map or
      the other. A throttled prune (run from the End callback, so it operates even while eviction is off)
      bounds the map by age; Guids are never reused.
  • De-duplicate the error telemetry. The full "Failed to find active enumeration ID" error is
    emitted once per enumeration ID within a window; repeats from a caller's retry loop are suppressed.
    The machine-based regression signal is preserved because the first occurrence still logs at Error.

Testing

  • WindowsFileSystemVirtualizerTests: 32/32 passed (targeted class run).
  • GetDirectoryEnumerationTagsMissReasonAndDeduplicates (replaces the prior
    ...TagsEvictedVersusUnknownId) asserts all three reasons - Evicted, NeverSeen,
    EndedRecently - and that repeated Gets for the same missing ID emit the error only once.

Cross-repo contract

The EnumerationFailureReason values are a case-sensitive string contract consumed by the workbook
in devprod.git.telemetry (gvfs-regression-signatures.kql + the cause-tag canary). That workbook's
expected enum set must add EndedRecently and NeverSeen (the canary tile will otherwise flag the
new values as unclassified). Companion workbook PR: EngSys 16223444.

Related

The EndedRecently race has a proposed kernel-side fix (enumeration rundown/drain in
gvflt dir.c/context.c) being prototyped separately in os.2020; this change makes the race
observable and quiet in the meantime.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

GetDirectoryEnumeration logs "Failed to find active enumeration ID" when an
enumeration ID is absent. Every non-eviction miss carried the reason Unknown,
which hid two different causes:

- EndedRecently: ProjFS delivered a Get that raced or followed the End for the
  same enumeration (a benign kernel close/query race).
- NeverSeen: GVFS never held the ID (it never started, or it predates a
  provider restart).

Record recently-ended IDs before removing them from activeEnumerations, so a
racing Get always finds the ID in one map or the other. Classify each miss as
Evicted, then EndedRecently, then NeverSeen. Rename the old Unknown value to
NeverSeen. Prune the tracking map on a throttle from EndDirectoryEnumeration so
it runs while eviction is off (the default).

De-duplicate the error: a caller that re-enumerates a lost handle can emit the
same error millions of times on one machine. Emit the full error once per ID
within a window. The machine-based signal stays intact because the first
occurrence still logs at Error. The returned HResult does not change; a status
change is behavior and belongs with the kernel-side fix.

The EnumerationFailureReason values match the bucketing in devprod.git.telemetry
(gvfs-regression-signatures.kql). The canary there must add EndedRecently and
NeverSeen.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tyrielv tyrielv closed this Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant