fix: IOException thrown issue from HtmlPostProcessor #10637

filzrev · 2025-04-17T10:28:19Z

This PR intended to fix #8654.
And contains following changes.

1. Add retry logics when HtmlPostProcessor throw IOException

Add retry logics to confirm behaviors. (Retry 3-times with delay ~~500ms/1000ms/1500ms~~ 0ms/500ms/1000ms)
I don't sure this retry resolve reported issues though.
But it make possible to isolate the problem. file locking problem is transient issue or not.

2. Add additional logging when IOException is thrown
Add logics to find processes that lock target file by using Win32 APIs. and add warnings.
See: https://github.com/dotnet/runtime/issues/109927 for details.

It can detect when file is locked by external process (e.g. Windows Defender) or locked by self process.

MartyIX · 2025-04-17T17:02:51Z

Thank you for working on this!

VaclavElias · 2025-04-19T12:49:58Z

Thanks for looking into this.

MartyIX · 2025-05-03T09:26:25Z

(Another try to generate my API doc and this time I disabled _gitContribute and it does not affect the result whatsoever. 2 successful attempts out of like 10.)

Copilot

Pull Request Overview

This PR is intended to address an IOException issue from the HtmlPostProcessor by adding retry logic and additional logging to diagnose file locking problems.

Introduces retry logic (3 attempts with increasing delays) for file creation in ManifestFileWriter.cs.
Adds a new FileLockCheck.cs file that leverages Win32 APIs to detect processes locking a file.
Adds a new warning code (LockedFile) to support the additional logging and updates the project file to allow unsafe code blocks.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/Docfx.Common/Loggers/WarningCodes.cs	New warning constant for file locking detection
src/Docfx.Common/FileLockCheck.cs	New file for detecting processes locking a file using Win32 APIs
src/Docfx.Common/FileAbstractLayer/ManifestFileWriter.cs	Implemented retry logic on file creation with added logging
src/Docfx.Common/Docfx.Common.csproj	Enabled unsafe code blocks

src/Docfx.Common/FileLockCheck.cs

Copilot · 2025-05-21T09:29:20Z

src/Docfx.Common/FileAbstractLayer/ManifestFileWriter.cs

+        Retry:
+            try
            {
-                Directory.CreateDirectory(
-                    Path.Combine(_manifestFolder, file.RemoveWorkingFolder().GetDirectoryPath()));
-                var result = File.Create(Path.Combine(_manifestFolder, file.RemoveWorkingFolder()));
+                var fileStream = File.Create(path);
                entry.LinkToPath = null;
-                return result;
+                return fileStream;
            }
-            else
+            catch (IOException e) when ((e.HResult & 0x0000FFFF) == 32) // ERROR_SHARING_VIOLATION: 0x80070020
            {
-                var path = Path.Combine(OutputFolder, file.RemoveWorkingFolder());
-                Directory.CreateDirectory(Path.GetDirectoryName(path));
-                var result = File.Create(path);
-                entry.LinkToPath = path;
-                return result;
+                // If retry failed 3 times. throw exception
+                if (++retryCount > 3)
+                    throw;
+
+                var sleepDelay = 500 * retryCount;
+
+                var message = FileLockCheck.GetLockingProcessNames(path);
+                if (string.IsNullOrEmpty(message))
+                    message = "File is locked by other process";
+
+                Logger.LogWarning($"{message}. Retry after {sleepDelay}[ms]", file: path, code: WarningCodes.Build.LockedFile);
+                Thread.Sleep(500 * retryCount);
+                goto Retry;


[nitpick] Consider refactoring the retry logic to use a loop structure instead of a goto statement to improve code readability and maintainability.

I know using goto statement is not recommended generally.
But it's more readable for simple retry patterns.
(And for complex retry scenario. It's better to use library (e.g. Polly) )

MartyIX · 2025-06-03T09:05:54Z

(I still desperately need this. I've been trying to generate my API doc for 9th consecutive time right now :))

edit: I noticed that the exception comes after a lot of "disposing" log messages:

97 uids are unresolved.
Normalizing all the object to weak type
Feeding options from template...
Updating href...
Applying system metadata to manifest...
Applying templates to 1579 model(s)...
Exporting xref map...
XRef map exported.
Disposing processor ManagedReferenceDocumentProcessor ...
Disposing build step ApplyOverwriteDocumentForMref ...
Disposing build step BuildManagedReferenceDocument ...
Disposing build step FillMetadata ...
Disposing build step FillReferenceInformation ...
Disposing build step SplitClassPageToMemberLevel ...
Disposing build step ValidateManagedReferenceDocumentMetadata ...
Disposing processor RestApiDocumentProcessor ...
Disposing build step ApplyOverwriteDocumentForRestApi ...
Disposing build step BuildRestApiDocument ...
Disposing build step ValidateRestApiDocumentMetadata ...
Disposing processor UniversalReferenceDocumentProcessor ...
Disposing build step ApplyOverwriteDocumentForUref ...
Disposing build step BuildUniversalReferenceDocument ...
Disposing build step FillReferenceInformation ...
Disposing processor TocDocumentProcessor ...
Disposing build step BuildTocDocument ...
Disposing processor ResourceDocumentProcessor ...
Disposing build step ValidateResourceMetadata ...
Disposing processor ConceptualDocumentProcessor ...
Disposing build step BuildConceptualDocument ...
Disposing build step ValidateConceptualDocumentMetadata ...
Disposing processor ApiPageDocumentProcessor ...
IOException: The process cannot access the file 'C:\project\documentation\api\something.html' because  
it is being used by another process.
  at SafeFileHandle CreateFile(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
  at SafeFileHandle Open(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
  at ctor(string path, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
  at Stream Create(RelativePath file) in ManifestFileWriter.cs:37
  at Stream Create(string file) in RootedFileAbstractLayer.cs:25
  at Manifest Process(Manifest manifest, string outputFolder, CancellationToken cancellationToken) in HtmlPostProcessor.cs:84
  at void Process(Manifest manifest, string outputFolder, CancellationToken cancellationToken) in PostProcessorsManager.cs:41
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs:149
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string templateDirectory, CancellationToken
     cancellationToken) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory, CancellationToken cancellationToken) in RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options, CancellationToken cancellationToken) in DefaultCommand.cs:31
  at int Execute(CommandContext context, TSettings settings) in CancellableCommandBase.cs:24
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:166

Perhaps it's related.

VaclavElias · 2025-06-03T10:09:24Z

I patiently keep generating 🤣. It's probably not an easy fix, I think there were some previous PRs trying address this. Luckily, it doesn't happen when I deploy through GitHub Actions, but yeah, while testing and working with docs on my PC, it gets annoying 🤣.

icnocop · 2025-07-16T23:52:06Z

src/Docfx.Common/FileAbstractLayer/ManifestFileWriter.cs

+
+                var message = FileLockCheck.GetLockingProcessNames(path);
+                if (string.IsNullOrEmpty(message))
+                    message = "File is locked by other process";


Suggested change

message = "File is locked by other process";

message = "File is locked by another process";

filzrev force-pushed the fix-issue-8654 branch from 10d73c9 to d80c7bc Compare April 17, 2025 10:33

filzrev force-pushed the fix-issue-8654 branch from d80c7bc to 0da7e3b Compare April 17, 2025 17:59

yufeih requested a review from Copilot May 21, 2025 09:28

Copilot AI reviewed May 21, 2025

View reviewed changes

filzrev force-pushed the fix-issue-8654 branch 2 times, most recently from 7464aff to 8101b80 Compare May 27, 2025 11:02

filzrev added 3 commits May 30, 2025 22:05

fix: IoException thrown issue at HtmlPostProcessor

3aaa093

chore: fix garbled characters

456992a

chore: retry immediately on first exception

b23961c

filzrev force-pushed the fix-issue-8654 branch from 8101b80 to b23961c Compare May 30, 2025 13:05

icnocop reviewed Jul 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: IOException thrown issue from HtmlPostProcessor #10637

fix: IOException thrown issue from HtmlPostProcessor #10637

Uh oh!

filzrev commented Apr 17, 2025 •

edited

Loading

Uh oh!

MartyIX commented Apr 17, 2025

Uh oh!

VaclavElias commented Apr 19, 2025

Uh oh!

MartyIX commented May 3, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI May 21, 2025

Uh oh!

filzrev May 21, 2025

Uh oh!

MartyIX commented Jun 3, 2025 •

edited

Loading

Uh oh!

VaclavElias commented Jun 3, 2025

Uh oh!

icnocop Jul 16, 2025

Uh oh!

Uh oh!

	message = "File is locked by other process";
	message = "File is locked by another process";

fix: IOException thrown issue from HtmlPostProcessor #10637

Are you sure you want to change the base?

fix: IOException thrown issue from HtmlPostProcessor #10637

Uh oh!

Conversation

filzrev commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MartyIX commented Apr 17, 2025

Uh oh!

VaclavElias commented Apr 19, 2025

Uh oh!

MartyIX commented May 3, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI May 21, 2025

Choose a reason for hiding this comment

Uh oh!

filzrev May 21, 2025

Choose a reason for hiding this comment

Uh oh!

MartyIX commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VaclavElias commented Jun 3, 2025

Uh oh!

icnocop Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

filzrev commented Apr 17, 2025 •

edited

Loading

MartyIX commented Jun 3, 2025 •

edited

Loading