Helix test runs fail with: The response contained an invalid status code 500 Internal Server Error

[jkotas]

Build

https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_build/results?buildId=1676838

Build leg reported

Mono llvmfullaot Pri0 Runtime Tests Run Linux x64 release / Send tests to Helix (Unix)

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "errorMessage" : ""
}

Additional information about the issue reported

No response

[dkurepa]

Hello, we are looking into this issue

[MattGal]

This was caused by transient DNS outage from the Helix API talking to the Azure storage resource provider ("No such host is known.")

Here's an App Insights query showing the exception

@dkurepa we can pretty safely add more retry logic in the /api/storage path, by tweaking the settings that are already used.

[alexperovich]

I am fairly certain there is already retry logic built in to this code path. I will check though.

[alexperovich]

Yep, there is already lots of retry logic built into this code path, adding more would just add more strain to the system. If this starts happening a significant amount then we can explore other options.