Tests fail with Microsoft.NET.Test.Sdk 18.8.0 due to timeout

[martincostello]

I've just received a number of renovate PRs to upgrade Microsoft.NET.Test.Sdk from 18.7.0 to 18.8.0, and they are failing with the following error which appears to be consistent across all operating systems:

Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.

Test Run Aborted.

I see the same locally on my computer, so this doesn't appear to be genuine slowness:

project-euler on  renovate/nuget/vstest-monorepo via .NET v10.0.301
❯ .\build.ps1
Building solution...
Restore complete (2.2s)
  ProjectEuler net10.0 succeeded (4.0s) → artifacts\bin\ProjectEuler\release\ProjectEuler.dll
  ProjectEuler.Benchmarks net10.0 succeeded (2.0s) → artifacts\bin\ProjectEuler.Benchmarks\release\ProjectEuler.Benchmarks.dll
  ProjectEuler.Tests net10.0 succeeded (1.8s) → artifacts\bin\ProjectEuler.Tests\release\ProjectEuler.Tests.dll

Build succeeded in 10.0s
Running tests...
Restore complete (0.5s)
  ProjectEuler net10.0 succeeded (0.4s) → artifacts\bin\ProjectEuler\release\ProjectEuler.dll
  ProjectEuler.Benchmarks net10.0 succeeded (0.3s) → artifacts\bin\ProjectEuler.Benchmarks\release\ProjectEuler.Benchmarks.dll
  ProjectEuler.Tests net10.0 succeeded (0.3s) → artifacts\bin\ProjectEuler.Tests\release\ProjectEuler.Tests.dll
Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
  ProjectEuler.Tests test net10.0 failed with 1 error(s) (90.4s)
    D:\Coding\martincostello\project-euler\artifacts\bin\ProjectEuler.Tests\release\ProjectEuler.Tests.dll : error TESTRUNABORT: Test Run Aborted.

Build failed with 1 error(s) in 92.2s
Exception: D:\Coding\martincostello\project-euler\build.ps1:92
Line |
  92 |          throw "dotnet test failed with exit code $LASTEXITCODE"
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | dotnet test failed with exit code 1

Example PRs:

• Bump dependency Microsoft.NET.Test.Sdk to 18.8.1 martincostello/api#3435
• Bump dependency Microsoft.NET.Test.Sdk to 18.8.1 martincostello/openapi-extensions#917
• Bump dependency Microsoft.NET.Test.Sdk to 18.8.1 martincostello/project-euler#925

The failure doesn't seem to be universal though, as this one succeeded: martincostello/lambda-test-server#1361

[nohwnd]

That was fast, looking.

[nohwnd]

Your projects disable reflection serialization by JsonSerializerIsReflectionEnabledByDefault=false, this will propagate to runtimeconfig that is then used by testhost and break our serializer. In 18.9 I think we are fixing this, I am looking into porting it back to 18.8 and also making it easier to see the actual error.

Is it possible for you to disable JsonSerializerIsReflectionEnabledByDefault=false for the testprojects and only keep them for product projects, temporarily?

[nohwnd]

Looks like there is 1 line fix for this, that would be safe to put into 18.8 (much smaller than the actual native aot fix done in 18.9).

[martincostello]

I could certainly do that temporarily to unblock the updates, but if they aren't critical I'd just leave them and take a patch with the fix instead.

Those projects intentionally disable reflection in the serializer as they use native AoT and I've had bugs in the past where the tests have passed but things have failed at runtime due to misuse of the serializer. Without it there's a coverage gap.

[nohwnd]

18.8.1 should come by the end of today.

[0xced]

Microsoft.NET.Test.Sdk 18.8.1 is there but its Microsoft.TestPlatform.TestHost 18.8.1 dependency still hasn't reached NuGet, rendering it unusable. 😕

error NU1102: Unable to find package Microsoft.TestPlatform.TestHost with version (>= 18.8.1)

[martincostello]

I think the release is just still in the process of publishing - I had similar errors with 18.8.0 and they self-resolved after a period.

[nohwnd]

The are all there now. Publish failed why I was in a meeting 🙄

[martincostello]

All PRs in the original issue description are working now with 18.8.1.