JIT tests - Unhandled IOException connection timed out

[carlossanlop]

Occurrences: (happens during random JIT tests)

Day	Run	OS	Details
9/21	Rolling run 25067 (7.0-rc2)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	Console
9/19	PR 23126 - #75675 (7.0-rc2)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	Console - reported in this issue originally
9/10	Reported in #75391 (7.0)	Mono Browser wasm Release @ Ubuntu.1804.Amd64.Open	JIT.HardwareIntrinsics.X86.Ssse3

Original report

This might be related to #75391 but it's the first time I see the failure message so I'm opening a new issue to be safe.

Need help confirming if related to the above or not, and also need help determining if this failure needs a fix backported to the 7.0-rc2 or the 7.0 branch.

Found in this release/7.0-rc2 deps update PR: #75675

• Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=22359&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=41
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75675-merge-b703d225d0044e25a7/JIT.PGO/1/console.51f35389.log?helixlogtype=result

Console log

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 7.0.0-rc.1.22423.16)
  Discovering: JIT.PGO.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.PGO.XUnitWrapper (found 4 test cases)
  Starting:    JIT.PGO.XUnitWrapper (parallel test collections = on, max threads = 2)
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh [FAIL]
      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-BSVTwDRbqiZ8pAK9bysU7lvA2UC4yhR_+Bc8sTbm8Jw'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
      
      Return code:      82
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.output.txt
      Raw output:
      BEGIN EXECUTION
      
      Welcome to .NET 7.0!
      ---------------------
      SDK Version: 7.0.100-rc.1.22431.11
      
      ----------------
      Installed an ASP.NET Core HTTPS development certificate.
      To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
      Learn about HTTPS: https://aka.ms/dotnet-https
      ----------------
      Write your first app: https://aka.ms/dotnet-hello-world
      Find out what's new: https://aka.ms/dotnet-whats-new
      Explore documentation: https://aka.ms/dotnet-docs
      Report issues and find source on GitHub: https://github.com/dotnet/core
      Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
      --------------------------------------------------------------------------------------
      
      Infra issue was detected and a work item retry was requested
      Test Harness Exitcode is : 82
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-BSVTwDRbqiZ8pAK9bysU7lvA2UC4yhR_+Bc8sTbm8Jw'
           at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
           at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
           at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
           at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
           at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
           at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
        
        Return code:      82
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.output.txt
        Raw output:
        BEGIN EXECUTION
        
        Welcome to .NET 7.0!
        ---------------------
        SDK Version: 7.0.100-rc.1.22431.11
        
        ----------------
        Installed an ASP.NET Core HTTPS development certificate.
        To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
        Learn about HTTPS: https://aka.ms/dotnet-https
        ----------------
        Write your first app: https://aka.ms/dotnet-hello-world
        Find out what's new: https://aka.ms/dotnet-whats-new
        Explore documentation: https://aka.ms/dotnet-docs
        Report issues and find source on GitHub: https://github.com/dotnet/core
        Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
        --------------------------------------------------------------------------------------
        
        Infra issue was detected and a work item retry was requested
        Test Harness Exitcode is : 82
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh [FAIL]
      
      Return code:      -100
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.output.txt
      Raw output:
      
      Work item retry had been requested earlier - skipping test...
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random1_ProfileCastClassAndIsInst_random1_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random1_ProfileCastClassAndIsInst_random1_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      -100
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.output.txt
        Raw output:
        
        Work item retry had been requested earlier - skipping test...
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh [FAIL]
      
      Return code:      -100
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.output.txt
      Raw output:
      
      Work item retry had been requested earlier - skipping test...
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random3_ProfileCastClassAndIsInst_random3_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random3_ProfileCastClassAndIsInst_random3_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      -100
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.output.txt
        Raw output:
        
        Work item retry had been requested earlier - skipping test...
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh
  Finished:    JIT.PGO.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.PGO.XUnitWrapper  Total: 4, Errors: 0, Failed: 3, Skipped: 0, Time: 9.458s

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This might be related to #75391 but it's the first time I see the failure message so I'm opening a new issue to be safe.

Need help confirming if related to the above or not, and also need help determining if this failure needs a fix backported to the 7.0-rc2 or the 7.0 branch.

Found in this release/7.0-rc2 deps update PR: #75675

• Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=22359&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=41
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75675-merge-b703d225d0044e25a7/JIT.PGO/1/console.51f35389.log?helixlogtype=result

Callstack

Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 7.0.0-rc.1.22423.16)
  Discovering: JIT.PGO.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.PGO.XUnitWrapper (found 4 test cases)
  Starting:    JIT.PGO.XUnitWrapper (parallel test collections = on, max threads = 2)
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh [FAIL]
      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-BSVTwDRbqiZ8pAK9bysU7lvA2UC4yhR_+Bc8sTbm8Jw'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
      
      Return code:      82
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.output.txt
      Raw output:
      BEGIN EXECUTION
      
      Welcome to .NET 7.0!
      ---------------------
      SDK Version: 7.0.100-rc.1.22431.11
      
      ----------------
      Installed an ASP.NET Core HTTPS development certificate.
      To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
      Learn about HTTPS: https://aka.ms/dotnet-https
      ----------------
      Write your first app: https://aka.ms/dotnet-hello-world
      Find out what's new: https://aka.ms/dotnet-whats-new
      Explore documentation: https://aka.ms/dotnet-docs
      Report issues and find source on GitHub: https://github.com/dotnet/core
      Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
      --------------------------------------------------------------------------------------
      
      Infra issue was detected and a work item retry was requested
      Test Harness Exitcode is : 82
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-BSVTwDRbqiZ8pAK9bysU7lvA2UC4yhR_+Bc8sTbm8Jw'
           at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
           at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
           at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
           at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
           at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
           at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
        
        Return code:      82
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.output.txt
        Raw output:
        BEGIN EXECUTION
        
        Welcome to .NET 7.0!
        ---------------------
        SDK Version: 7.0.100-rc.1.22431.11
        
        ----------------
        Installed an ASP.NET Core HTTPS development certificate.
        To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
        Learn about HTTPS: https://aka.ms/dotnet-https
        ----------------
        Write your first app: https://aka.ms/dotnet-hello-world
        Find out what's new: https://aka.ms/dotnet-whats-new
        Explore documentation: https://aka.ms/dotnet-docs
        Report issues and find source on GitHub: https://github.com/dotnet/core
        Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
        --------------------------------------------------------------------------------------
        
        Infra issue was detected and a work item retry was requested
        Test Harness Exitcode is : 82
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst.sh
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh [FAIL]
      
      Return code:      -100
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.output.txt
      Raw output:
      
      Work item retry had been requested earlier - skipping test...
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random1_ProfileCastClassAndIsInst_random1_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random1_ProfileCastClassAndIsInst_random1_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      -100
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.output.txt
        Raw output:
        
        Work item retry had been requested earlier - skipping test...
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random1/ProfileCastClassAndIsInst_random1.sh
    JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh [FAIL]
      
      Return code:      -100
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.output.txt
      Raw output:
      
      Work item retry had been requested earlier - skipping test...
      Test Harness Exitcode is : -100
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_PGO._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random3_ProfileCastClassAndIsInst_random3_._ProfileCastClassAndIsInst_ProfileCastClassAndIsInst_random3_ProfileCastClassAndIsInst_random3_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      -100
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B34F09D3/uploads/Reports/JIT.PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.output.txt
        Raw output:
        
        Work item retry had been requested earlier - skipping test...
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B34F09D3/e/JIT/PGO/ProfileCastClassAndIsInst/ProfileCastClassAndIsInst_random3/ProfileCastClassAndIsInst_random3.sh
  Finished:    JIT.PGO.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.PGO.XUnitWrapper  Total: 4, Errors: 0, Failed: 3, Skipped: 0, Time: 9.458s

Author:	carlossanlop
Assignees:	-
Labels:	arch-wasm, os-linux, area-VM-meta-mono
Milestone:	-

[carlossanlop]

@rainersigwald / @rokonec / @AR-May / @MichalPavlik this is happening in a few PRs in release/7.0-rc2.

We found it in this other PR: #75885

[rainersigwald]

cc @marcpopMSFT for risk on MSBuild Server.

[carlossanlop]

I just noticed that among the several different failures reported in this other general JIT issue, this one was already reported there: #75391

I'm not sure if we can consider all the failure messages related to the exact same root cause, but I wanted to mention the other issue in case we need to close this one as duplicate.

cc @radical who opened the other one.

[MichalPavlik]

Based on the log file/MSBuild stack trace, I would say that the root cause is same as for #75391. I will create issue in MSBuild repo.

[rokonec]

@carlossanlop The root cause of this bug is not obvious from logs. Is there a way we can somehow repro it locally? Or at least run it in CI as debug so we have more precise exception stack?

[lewing]

closing as a duplicate of #75391 (which appears to have resolved by disabling the compilation server)