Split enumeration-miss cause (EndedRecently vs NeverSeen) + de-duplicate error telemetry - #9
Closed
tyrielv wants to merge 1 commit into
Closed
Conversation
GetDirectoryEnumeration logs "Failed to find active enumeration ID" when an enumeration ID is absent. Every non-eviction miss carried the reason Unknown, which hid two different causes: - EndedRecently: ProjFS delivered a Get that raced or followed the End for the same enumeration (a benign kernel close/query race). - NeverSeen: GVFS never held the ID (it never started, or it predates a provider restart). Record recently-ended IDs before removing them from activeEnumerations, so a racing Get always finds the ID in one map or the other. Classify each miss as Evicted, then EndedRecently, then NeverSeen. Rename the old Unknown value to NeverSeen. Prune the tracking map on a throttle from EndDirectoryEnumeration so it runs while eviction is off (the default). De-duplicate the error: a caller that re-enumerates a lost handle can emit the same error millions of times on one machine. Emit the full error once per ID within a window. The machine-based signal stays intact because the first occurrence still logs at Error. The returned HResult does not change; a status change is behavior and belongs with the kernel-side fix. The EnumerationFailureReason values match the bucketing in devprod.git.telemetry (gvfs-regression-signatures.kql). The canary there must add EndedRecently and NeverSeen. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on microsoft#2071 (base branch
tyrielv/split-hydration-enum-telemetry). This extends thedirectory-enumeration cause tagging with a finer split and removes a telemetry noise source. No
behavior change - metadata and log-volume only; the returned
HResultis unchanged.Why
On the current LKG (1.0.26014.1), "Directory enumeration failure" ("Failed to find active
enumeration ID") is the largest named regression signature: ~9.7% of os.2020 mount machines hit it
in 30d. microsoft#2071 tags the miss as
EvictedvsUnknown. Field analysis of thatUnknownbucket showstwo distinct causes that need to be told apart, plus a per-machine event storm that inflates the
signal:
restartScan == false; 96% occur >60 min after mount (steady state, not astartup/projection-not-ready race).
GetDirectoryEnumerationthat races theEndDirectoryEnumerationfor the same handle - a query in flight while the directory handle isclosing, or the querying process dying mid-enumeration. This is a benign close/query race, not an
ID GVFS never held.
re-issue the same failing Get in a tight loop; a single stuck enumeration has produced millions of
error events on one machine, drowning the machine-based signal.
Changes
Split the miss reason.
EnumerationFailureReasonrenamesUnknown->NeverSeenand addsEndedRecently:Evicted- GVFS's own stale-enumeration eviction removed a live enumeration (self-inflicted).EndedRecently- a Get raced or followed the End for the same enumeration (benign kernelclose/query race; outside gvfs.exe's control).
NeverSeen- GVFS never held the ID: never started, or from before a provider restart (outsidegvfs.exe's control).
EndDirectoryEnumerationCallbackrecords the ended ID in arecentlyEndedEnumerationsmapbefore removing it from
activeEnumerations, so a racing Get always finds the ID in one map orthe other. A throttled prune (run from the End callback, so it operates even while eviction is off)
bounds the map by age; Guids are never reused.
De-duplicate the error telemetry. The full "Failed to find active enumeration ID" error is
emitted once per enumeration ID within a window; repeats from a caller's retry loop are suppressed.
The machine-based regression signal is preserved because the first occurrence still logs at Error.
Testing
WindowsFileSystemVirtualizerTests: 32/32 passed (targeted class run).GetDirectoryEnumerationTagsMissReasonAndDeduplicates(replaces the prior...TagsEvictedVersusUnknownId) asserts all three reasons -Evicted,NeverSeen,EndedRecently- and that repeated Gets for the same missing ID emit the error only once.Cross-repo contract
The
EnumerationFailureReasonvalues are a case-sensitive string contract consumed by the workbookin
devprod.git.telemetry(gvfs-regression-signatures.kql+ the cause-tag canary). That workbook'sexpected enum set must add
EndedRecentlyandNeverSeen(the canary tile will otherwise flag thenew values as unclassified). Companion workbook PR: EngSys 16223444.
Related
The
EndedRecentlyrace has a proposed kernel-side fix (enumeration rundown/drain ingvflt
dir.c/context.c) being prototyped separately in os.2020; this change makes the raceobservable and quiet in the meantime.
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com