System.Net.WebSockets.Client.Tests crash on OSX Mono

[karelz]

Frequency in last 30 days in Runfo:

• 9/8 - 33x failures
  • First occurrence in Rolling run 3461 on 9/2 -- recent regression
  • Each run fails on 2 platforms:
    • net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1200.Arm64.Open
    • net7.0-MacCatalyst-Release-x64-Mono_Release-OSX.1200.Amd64.Open
    • Note: There's is most likely unrelated failure on net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open in each of these runs

Failure:

[15:49:31] dbug: Test execution started
[15:49:32] dbug: Test log server listening on: 0.0.0.0:62555
[15:49:32] dbug: Process open exited with 0
[15:49:42] dbug: Test run completed
[15:49:42] dbug: Process log exited with 137
[15:49:42] dbug: Test run started but crashed and no test results were reported
[15:49:42] dbug: No crash reports, waiting 30 seconds for the crash report service...
[15:49:42] dbug: Found 1 new crash report(s)
[15:49:42] dbug: - Adding /Users/helix-runner/Library/Logs/DiagnosticReports/System.Net.WebSockets.Client.Tests-2022-09-02-154941.ips
[15:49:42] dbug: Successfully copied System.Net.WebSockets.Client.Tests-2022-09-02-154941.ips
[15:49:42] fail: Application test run crashed
                 No test log file was produced
                 
                 Check logs for more information
[15:49:42] dbug: Saving diagnostics data to '/tmp/helix/working/9A9B08C3/w/AC6B0966/e/diagnostics.json'
XHarness exit code: 80 (APP_CRASH)

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Frequency in last 30 days in Runfo:

• 9/8 - 33x failures
  • First occurrence in Rolling run 3461 on 9/2 -- recent regression
  • Each run fails on 2 platforms:
    • net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1200.Arm64.Open
    • net7.0-MacCatalyst-Release-x64-Mono_Release-OSX.1200.Amd64.Open
    • Note: There's is most likely unrelated failure on net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open in each of these runs

Author:	karelz
Assignees:	-
Labels:	area-System.Net, blocking-clean-ci, runtime-mono, os-maccatalyst
Milestone:	-

[karelz]

@akoeplinger @steveisok this is blocking clean CI -- happens 5x per day. Can you please look into it ASAP? Thanks!

[steveisok]

Looks like it's an mlaunch issue that can't start the app. We'll need to figure out how widespread this is.

[steveisok]

@mandel-macaque We haven't bumped xharness/mlaunch in runtime yet. Is this an mlaunch problem or with devices?

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9747453e45bd47beab/System.Net.WebSockets.Client.Tests/1/console.83d2be01.log?helixlogtype=result

[14:58:25] dbug: [TCP tunnel] Xamarin.Hosting: Device discovery started
[14:58:25] dbug: [TCP tunnel] Xamarin.Hosting: Device discovery event: Connected (fff44:c6:5d:6d:09:3afff)
[14:58:25] dbug: [TCP tunnel] 2022-09-02 14:58:25.562 mlaunch[56071:1486223]
[14:58:25] dbug: [TCP tunnel] Unhandled Exception:
[14:58:25] dbug: [TCP tunnel] Xamarin.Hosting.MobileDeviceException: Read Error (error: 0xe8000004)
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.RealDevice.Connect () [0x00021] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:57
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.RealDevice..ctor (System.IntPtr am_device) [0x00022] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/RealDevice.cs:40
[14:58:25] dbug: [TCP tunnel]   at Xamarin.Hosting.DeviceDiscovery+<>c__DisplayClass23_1.<DiscoverNotificationCallback>b__0 () [0x00007] in /Users/builder/azdo/_work/3/s/tools/mlaunch/Xamarin.Hosting/Xamarin.Hosting/DeviceDiscovery.cs:225
[14:58:25] dbug: [TCP tunnel]   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in 

[karelz]

Note: It happens also in 7.0 branch -- Rolling run 9064 on 9/8 AM (release/7.0) ... so perhaps it is something environmental? Or some shared infra?

[akoeplinger]

The MacCatalyst failures shouldn't be related to mlaunch since afaik we don't use mlaunch to run Catalyst apps.

[mandel-macaque]

As @akoeplinger mentioned, mlaunch is not needed for maccatalyst, since it does not need to be added to a device neither it needs the TCP tunnel. Yet, the stack trace from @steveisok points to an issue when creating the tcp tunnel with an apple tv.

@steveisok there area few things we need to clarify what exactly is happening.

1. As per logs, there are two different apple tv devices: DNCENGTVOS-091 and DNCENGTVOS-061. Can we have some more info of what are the versions of those devices?
2. The issue happens when trying to create the tcp tunnel with 3315900bedf22783bd9d2a74d012464b00e0fc99 which is device DNCENGTVOS-061. We need someone to go to the lab and tell use the state of that device. Can we repo the issue with DNCENGTVOS-091 ?
3. The Xcode version used is Xcode13.app yet the latest stable is 13.4.1 (13F100) we need to verify the version of Xcode is used in the lab. mlaunch has had changes due to issues with the latests Xcodes.
4. I don't have access to the crash report. Can it be added to the issue?

Lets try to get answer to the above to keep looking into issues.