GenerateBundle Task fails writing to singlefilehost.exe

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1146095
Build error leg or test failing: Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution
Pull request: #50648

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "The process cannot access the file '.*singlefilehost\\.exe' because it is being used by another process",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1146095
Error message validated: [The process cannot access the file '.*singlefilehost\.exe' because it is being used by another process]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/12/2025 3:27:55 PM UTC

Report

Build	Definition	Test	Pull Request
1182892	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51425
1182766	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51425
1176247	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51299
1176130	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51293
1176102	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51264
1176061	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51214
1175890	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51214
1175651	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51280
1175361	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51274
1175246	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51269
1173188	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51143
1174082	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51209
1173688	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51207
1172371	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#50468
1171481	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51209
1170558	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51153
1169937	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#50946
1169494	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#50985
1169201	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51108
1168937	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51143
1168623	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51156
1168500	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51169
1168111	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51143
1167999	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51153
1167880	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51156
1167754	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51153
1167157	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51074
1167495	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51052
1167156	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51036
1167138	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51036
1167106	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51143
1166889	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51138
1166893	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#50980
1166850	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51088
1166699	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51074
1166526	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51135
1166457	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51119
1166108	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51134
1165607	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51127
1165384	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51074
1165036	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#48918
1165008	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51075
1165014	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51036
1164810	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51120
1164821	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51108
1164627	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51118
1164637	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51119
1164469	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51117
1164366	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51074
1163714	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51050
1163547	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51052
1163599	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51108
1163532	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51050
1163331	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51036
1163197	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51054
1163200	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51094
1163206	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51095
1162428	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51052
1163029	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51073
1159610	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51020

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	60

[dsplaisted]

@dotnet/illink @agocke We are seeing failures where the GenerateBundle task is failing because it cannot access the singlnefilehost.exe file. We added retry logic because we thought maybe the issues were caused by defender. I thought that helped, but now the issue is happening pretty consistently, so it may be a product issue.

Interestingly, it seems to only fail on Mac OS and Linux when bundling a Windows app. I can't remember if it was failing on Windows before we added the retry logic or not.

Can someone investigate this?

I'm also wondering whether we should merge this codeflow PR which is blocked by the failure. That's where I've seen the failure most consistently recently, so it's possible there's a code change coming in that made the problem worse.

[agocke]

@jtschuster can you take a look?

[dsplaisted]

Ping @jtschuster, any progress on this? I just noticed it in #50876

[agocke]

I'm going to load balance a bit. @sbomer can you look instead?

[jtschuster]

Still investigating, but I am able to repro locally on a mac, but only on the first run of the test. Subsequent runs use the same path for the tmp project and don't repro. It's also interesting that it's failing on a read and not a write.

[dsplaisted]

@jtschuster Any update on this? This is impacting a bunch of our CI runs.

[jtschuster]

Still investigating exactly what's going on, but my hunch is that there's an async memory map flush that keeps the apphost mapped from CreateAppHost. In particular it looks like this todo might fix the issue: https://github.com/dotnet/runtime/blob/b8c39cc4d102518f8608df70e6e3cb7a5c10d48d/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L172
The ResourceUpdater.Update() maps the destination apphost back into memory, and if there's async flushing after Dispose(), we could run into issues.

As a workaround, it looks like adding --disable-build-servers to the PublishCommand.Execute() method does something to make the issue fail to repro for both me and @sbomer. I'm not sure why, though.