Error reported in diagnostic logs. Please examine the log for more details

[jeffschwMSFT]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532

Error:

System.Net.Http.HttpRequestException: No route to host (dev.azure.com:443)
 ---> System.Net.Sockets.SocketException (65): No route to host
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.TaskServer.ConnectAsync(VssConnection jobConnection) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/TaskServer.cs:line 42
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 224
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 397
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 97
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52

Error reported in diagnostic logs. Please examine the log for more details

Build leg reported

No response

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "Error reported in diagnostic logs. Please examine the log for more details",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532
Error message validated: Error reported in diagnostic logs. Please examine the log for more details
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/1/2024 3:50:06 PM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
2995946	dotnet-maui	Build Device Tests (Mono)
2994375	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
1450721	dotnet/aspnetcore	Build: macOS arm64		dotnet/aspnetcore#67024
2992556	dotnet-maui	Build Device Tests (Mono)
2992237	dotnet-maui	Controls (v18.5) ScrollView	Log
2990306	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2989435	dotnet-maui	RunOniOS_MauiNativeAOT x64
2988506	dotnet-maui	Build Device Tests (Mono)
2987451	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2986146	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2983764	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2983768	dotnet-maui	Build Device Tests (Mono)
2983212	dotnet-maui	Build Device Tests (Mono)
2982050	dotnet-maui	Controls (vlatest) RadioButton,RefreshView	Log
2981651	dotnet-maui	Build Device Tests (Mono)
2980810	dotnet-maui	Build Device Tests (Mono)
2979788	dotnet-maui	Build Device Tests (Mono)
2979449	dotnet-maui	Controls (vlatest) SearchBar,Shape,Slider	Log
2979159	dotnet-maui	Build Device Tests (Mono)
2978797	dotnet-runtime	Libraries SuperPMI collection libraries_tests_no_tiered_compilation Checked coreclr osx arm64 Release	Log
2977978	dotnet-maui	Controls (v18.5)	Log
2971911	dotnet-maui	Controls (v18.5)	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	5	22

[lewing]

Looks like mostly macs? not sure where Build Source Index runs

[epananth]

Looked at the latest builds all are failing cos out of space on the device

[epananth]

##[error]Error reported in diagnostic logs. Please examine the log for more details. - /Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log System.IO.IOException: No space left on device : '/Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 151
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 81
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 93
at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52`

[epananth]

I am going to go ahead an add a check for disk utilization, to know more why this is failing.

[epananth]

Is it possible to add this check to your pipeline to get more data (like aspnet-core team did)

https://github.com/dotnet/aspnetcore/blob/42af9fe6ddd7c3f9cde04ac003bf97509881873b/.azure/pipelines/jobs/default-build.yml#L163-L165

[missymessa]

@jeffschwMSFT @lewing is this still an issue?

[lewing]

@jeffschwMSFT @lewing is this still an issue?

yes

[lewing]

I took a look at the recent entries in this issue. All the latest builds hitting this error are from the dotnet-runtime-perf and dotnet-runtime-perf-build pipelines running on NetCore1ESPool-Internal agents.

Checking the build logs (e.g. builds 2926621, 2926620, 2926505, 2926495, 2926451, 2926370), they all show the same root cause: disk space exhaustion on the agents — consistently at 97–100% usage. This triggers a cascade of failures:

• Docker pulls fail with "no space left on device"
• Build processes crash with SIGABRT (exit code 134)
• Timeouts as processes stall on disk I/O
• The generic "Error reported in diagnostic logs" wrapper error

This appears to be the same underlying issue reported by @epananth back in Aug 2024 — the perf build agents need either larger disks or more aggressive cleanup between jobs. The disk utilization check that was added helps surface the problem, but the root cause (insufficient disk capacity for the workload) is still present.

[lewing]

cc @DrewScoggins @mangod9

[DrewScoggins]

Yeah, we are seeing this as well. We have a PR up, dotnet/runtime#125634, to fix this on the build side, and should have another up to fix the perf job side as well.