[7.0] [tvOSSimulator] Ran out of trampolines of type 1 in System.Private.CoreLib

[carlossanlop]

Error Blob

{
  "ErrorMessage": "Ran out of trampolines of type 1",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• PR: [automated] Merge branch 'release/7.0' => 'release/7.0-staging' #88652
• Affected test: System.Text.Json.Serialization.Tests.StreamTests_Sync.LargeJsonFile cc @dotnet/area-system-text-json
• Queue: Build tvOSSimulator x64 Release AllSubsets_Mono
• Run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=336356&view=logs&j=e4e455c5-b032-5051-d2c5-a64982c9ac45&t=5de5e156-699a-557b-38b4-10fd41c307dc
• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-8cc8ee158be14a129f/System.Text.Json.Tests.Attempt.3/1/console.341a6769.log?helixlogtype=result
• Callstack:

[01:42:25] dbug: 2023-07-12 01:42:25.843865-0400 System.Text.Json.Tests[93349:4705603] ((null) error) Ran out of trampolines of type 1 in '/Users/helix-runner/Library/Developer/CoreSimulator/Devices/7D1C8F1B-ADD1-43AC-A119-B9E3C4362131/data/Containers/Bundle/Application/D14F1382-F955-4545-92F4-A7FFEDB574E3/System.Text.Json.Tests.app/System.Private.CoreLib.dll' (limit 31000)
[01:42:25] dbug: 2023-07-12 01:42:25.843969-0400 System.Text.Json.Tests[93349:4705603] DOTNET.APP_EXIT_CODE: 1
[01:42:25] dbug: Test log server listening on: 0.0.0.0:50054
[01:42:26] dbug: Xamarin.Hosting: Simulated process has exited (could not get process info: Can't find process with ID 93349).
[01:42:26] dbug: Process mlaunch exited with 0
[01:42:26] dbug: Test run completed
[01:42:26] dbug: Test run started but crashed and no test results were reported
[01:42:26] dbug: No crash reports, waiting 30 seconds for the crash report service...
[01:42:57] fail: Application test run crashed
                 No test log file was produced
                 
                 Check logs for more information
[01:42:57] info: Uninstalling the application 'net.dot.System.Text.Json.Tests' from 'Apple TV'
[01:42:57] dbug: 
[01:42:57] dbug: Running /Applications/Xcode_13.1.app/Contents/Developer/usr/bin/simctl uninstall 7D1C8F1B-ADD1-43AC-A119-B9E3C4362131 net.dot.System.Text.Json.Tests
[01:42:58] dbug: Process simctl exited with 0
[01:42:58] info: Application 'net.dot.System.Text.Json.Tests' was uninstalled successfully
[01:42:58] dbug: Saving diagnostics data to '/tmp/helix/working/B5C909A3/w/9B120850/e/diagnostics.json'
XHarness exit code: 80 (APP_CRASH)
Removing empty log files:
ls: /tmp/helix/working/B5C909A3/w/9B120850/uploads/xunit-*.xml: No such file or directory
+ exit_code=80
+ '[' -f /tmp/helix/working/B5C909A3/w/9B120850/e/.retry ']'
+ '[' -f /tmp/helix/working/B5C909A3/w/9B120850/e/.reboot ']'
+ exit 80
+ export _commandExitCode=80
+ _commandExitCode=80
+ /usr/local/bin/python3 -u /tmp/helix/working/B5C909A3/w/9B120850/u/xharness-event-processor.py
Reporting 1 events from diagnostics file `/tmp/helix/working/B5C909A3/w/9B120850/e/diagnostics.json`
Analyzing apple/test@tvos-simulator (80)
    Application crashed - if persist, please investigate system logs from the run
2023-07-12T05:43:00.206Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.288Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.288Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.361Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.361Z	INFO   	xharness-event-processor.py	workitemutil(23)	request_infra_retry	Infrastructure retry (rerun of work item) has been requested
2023-07-12T05:43:00.362Z	INFO   	xharness-event-processor.py	workitemutil(47)	_write_workitem_status_file	Created file 'infraretry-1a3d1002-f526-4355-952f-515ce9301966-8cc8ee15-8be1-4a12-9f18-ec210bf5085c.txt' to indicate a request for infraretry
2023-07-12T05:43:00.362Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.430Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.430Z	INFO   	xharness-event-processor.py	workitemutil(14)	request_reboot	Post-workitem reboot has been requested.
2023-07-12T05:43:00.431Z	INFO   	xharness-event-processor.py	workitemutil(47)	_write_workitem_status_file	Created file 'reboot-1a3d1002-f526-4355-952f-515ce9301966-8cc8ee15-8be1-4a12-9f18-ec210bf5085c.txt' to indicate a request for reboot
+ /usr/local/bin/python3 /tmp/helix/working/B5C909A3/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 6987804 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJlZGNiNWFjYS0zYjMxLTRkMWMtOTJiOC0wMzQ5Mjk1MWNlNDgiLCJzaWQiOiI4OWI3NDhlOS04NGYzLTRkNDktOGEwNC1iNDcxNGNjNTY2NGYiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzMzNjM1NiIsImpvYnJlZiI6IjE5YTk0MWZhLTkzODYtNDhjNi05MzQ5LWVjZjFhNzVlZTlkNTplNGU0NTVjNS1iMDMyLTUwNTEtZDJjNS1hNjQ5ODJjOWFjNDUiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMzM2MzU2Iiwib3JjaGlkIjoiMTlhOTQxZmEtOTM4Ni00OGM2LTkzNDktZWNmMWE3NWVlOWQ1LmJ1aWxkX3R2b3NzaW11bGF0b3JfeDY0X3JlbGVhc2VfYWxsc3Vic2V0c19tb25vLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjg5MTMxNTM4LCJleHAiOjE2ODkxNDM1Mzh9.sjCOc4HCrBFNtMNdOp9iWsLRVzUktW7IJbNaEjMMcMObTZuWblkMo7OqGmjJYeOgZpw421UEPMMM4dfGa88feP1B0j_o6g6Ka1GaikFnuDFaBQ7P_-PMUOdBgRfdpWna1PefYZjcb8hEhInLWoKDduEGHyrzd2YFFZNgh5KbWJcCBNquBgACLTu4eVOUKzQmU9JAFm2-hA_TeohWIABwzSN8QF_EcRjuvAi25OPQREmzR4VeuNJarpi8UMppbqc-0pdto0zr8F7G3QWl7oV48YOtflN9ivAoQJUpWtVR728m9gL5gnSDLN0_yZj49Pmq5HU7yDviqtEycI0I2ivvxg
2023-07-12T05:43:01.096Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-07-12T05:43:01.098Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/B5C909A3/w/9B120850/e' for test results files
2023-07-12T05:43:01.100Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/B5C909A3/w/9B120850/uploads' for test results files
2023-07-12T05:43:01.100Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-07-12T05:43:01.101Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/B5C909A3/w/9B120850/e/__test_report.json'
2023-07-12T05:43:01.102Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1541 bytes
+ exit 80
./xharness-runner.apple.sh: line 173: 93316 Killed: 9               sleep "$command_timeout"
['System.Text.Json.Tests' END OF WORK ITEM LOG: Command exited with 80]

Report

Build	Definition	Test	Pull Request
341300	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#88854
341726	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#88723
341298	dotnet/runtime	System.Linq.Expressions.Tests.WorkItemExecution	#88723
339891	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
339787	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution	#88854
338978	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
337931	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
336908	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution
336356	dotnet/runtime	System.Text.Json.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	9

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=336356
Error message validated: Ran out of trampolines of type 1
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 7/12/2023 4:06:56 PM UTC

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

{
  "ErrorMessage": "Ran out of trampolines of type 1",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

• PR: [automated] Merge branch 'release/6.0' => 'release/6.0-staging' #88653
• Affected test: System.Text.Json.Serialization.Tests.StreamTests_Sync.LargeJsonFile cc @dotnet/area-system-text-json
• Queue: Build tvOSSimulator x64 Release AllSubsets_Mono
• Run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=336356&view=logs&j=e4e455c5-b032-5051-d2c5-a64982c9ac45&t=5de5e156-699a-557b-38b4-10fd41c307dc
• Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-8cc8ee158be14a129f/System.Text.Json.Tests.Attempt.3/1/console.341a6769.log?helixlogtype=result
• Callstack:

[01:42:25] dbug: 2023-07-12 01:42:25.843865-0400 System.Text.Json.Tests[93349:4705603] ((null) error) Ran out of trampolines of type 1 in '/Users/helix-runner/Library/Developer/CoreSimulator/Devices/7D1C8F1B-ADD1-43AC-A119-B9E3C4362131/data/Containers/Bundle/Application/D14F1382-F955-4545-92F4-A7FFEDB574E3/System.Text.Json.Tests.app/System.Private.CoreLib.dll' (limit 31000)
[01:42:25] dbug: 2023-07-12 01:42:25.843969-0400 System.Text.Json.Tests[93349:4705603] DOTNET.APP_EXIT_CODE: 1
[01:42:25] dbug: Test log server listening on: 0.0.0.0:50054
[01:42:26] dbug: Xamarin.Hosting: Simulated process has exited (could not get process info: Can't find process with ID 93349).
[01:42:26] dbug: Process mlaunch exited with 0
[01:42:26] dbug: Test run completed
[01:42:26] dbug: Test run started but crashed and no test results were reported
[01:42:26] dbug: No crash reports, waiting 30 seconds for the crash report service...
[01:42:57] fail: Application test run crashed
                 No test log file was produced
                 
                 Check logs for more information
[01:42:57] info: Uninstalling the application 'net.dot.System.Text.Json.Tests' from 'Apple TV'
[01:42:57] dbug: 
[01:42:57] dbug: Running /Applications/Xcode_13.1.app/Contents/Developer/usr/bin/simctl uninstall 7D1C8F1B-ADD1-43AC-A119-B9E3C4362131 net.dot.System.Text.Json.Tests
[01:42:58] dbug: Process simctl exited with 0
[01:42:58] info: Application 'net.dot.System.Text.Json.Tests' was uninstalled successfully
[01:42:58] dbug: Saving diagnostics data to '/tmp/helix/working/B5C909A3/w/9B120850/e/diagnostics.json'
XHarness exit code: 80 (APP_CRASH)
Removing empty log files:
ls: /tmp/helix/working/B5C909A3/w/9B120850/uploads/xunit-*.xml: No such file or directory
+ exit_code=80
+ '[' -f /tmp/helix/working/B5C909A3/w/9B120850/e/.retry ']'
+ '[' -f /tmp/helix/working/B5C909A3/w/9B120850/e/.reboot ']'
+ exit 80
+ export _commandExitCode=80
+ _commandExitCode=80
+ /usr/local/bin/python3 -u /tmp/helix/working/B5C909A3/w/9B120850/u/xharness-event-processor.py
Reporting 1 events from diagnostics file `/tmp/helix/working/B5C909A3/w/9B120850/e/diagnostics.json`
Analyzing apple/test@tvos-simulator (80)
    Application crashed - if persist, please investigate system logs from the run
2023-07-12T05:43:00.206Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.288Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.288Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.361Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.361Z	INFO   	xharness-event-processor.py	workitemutil(23)	request_infra_retry	Infrastructure retry (rerun of work item) has been requested
2023-07-12T05:43:00.362Z	INFO   	xharness-event-processor.py	workitemutil(47)	_write_workitem_status_file	Created file 'infraretry-1a3d1002-f526-4355-952f-515ce9301966-8cc8ee15-8be1-4a12-9f18-ec210bf5085c.txt' to indicate a request for infraretry
2023-07-12T05:43:00.362Z	INFO   	xharness-event-processor.py	event(51)	send	Sending event type _MobileDeviceOperation
2023-07-12T05:43:00.430Z	INFO   	xharness-event-processor.py	saferequests(87)	request_with_retry	Response complete with status code '201'
2023-07-12T05:43:00.430Z	INFO   	xharness-event-processor.py	workitemutil(14)	request_reboot	Post-workitem reboot has been requested.
2023-07-12T05:43:00.431Z	INFO   	xharness-event-processor.py	workitemutil(47)	_write_workitem_status_file	Created file 'reboot-1a3d1002-f526-4355-952f-515ce9301966-8cc8ee15-8be1-4a12-9f18-ec210bf5085c.txt' to indicate a request for reboot
+ /usr/local/bin/python3 /tmp/helix/working/B5C909A3/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 6987804 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJlZGNiNWFjYS0zYjMxLTRkMWMtOTJiOC0wMzQ5Mjk1MWNlNDgiLCJzaWQiOiI4OWI3NDhlOS04NGYzLTRkNDktOGEwNC1iNDcxNGNjNTY2NGYiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzMzNjM1NiIsImpvYnJlZiI6IjE5YTk0MWZhLTkzODYtNDhjNi05MzQ5LWVjZjFhNzVlZTlkNTplNGU0NTVjNS1iMDMyLTUwNTEtZDJjNS1hNjQ5ODJjOWFjNDUiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMzM2MzU2Iiwib3JjaGlkIjoiMTlhOTQxZmEtOTM4Ni00OGM2LTkzNDktZWNmMWE3NWVlOWQ1LmJ1aWxkX3R2b3NzaW11bGF0b3JfeDY0X3JlbGVhc2VfYWxsc3Vic2V0c19tb25vLl9fZGVmYXVsdCIsInJlcG9JZHMiOiIiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286NmZjYzkyZTUtNzNhNy00Zjg4LThkMTMtZDkwNDViNDVmYjI3IiwibmJmIjoxNjg5MTMxNTM4LCJleHAiOjE2ODkxNDM1Mzh9.sjCOc4HCrBFNtMNdOp9iWsLRVzUktW7IJbNaEjMMcMObTZuWblkMo7OqGmjJYeOgZpw421UEPMMM4dfGa88feP1B0j_o6g6Ka1GaikFnuDFaBQ7P_-PMUOdBgRfdpWna1PefYZjcb8hEhInLWoKDduEGHyrzd2YFFZNgh5KbWJcCBNquBgACLTu4eVOUKzQmU9JAFm2-hA_TeohWIABwzSN8QF_EcRjuvAi25OPQREmzR4VeuNJarpi8UMppbqc-0pdto0zr8F7G3QWl7oV48YOtflN9ivAoQJUpWtVR728m9gL5gnSDLN0_yZj49Pmq5HU7yDviqtEycI0I2ivvxg
2023-07-12T05:43:01.096Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2023-07-12T05:43:01.098Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/B5C909A3/w/9B120850/e' for test results files
2023-07-12T05:43:01.100Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/B5C909A3/w/9B120850/uploads' for test results files
2023-07-12T05:43:01.100Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2023-07-12T05:43:01.101Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/B5C909A3/w/9B120850/e/__test_report.json'
2023-07-12T05:43:01.102Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1541 bytes
+ exit 80
./xharness-runner.apple.sh: line 173: 93316 Killed: 9               sleep "$command_timeout"
['System.Text.Json.Tests' END OF WORK ITEM LOG: Command exited with 80]

Author:	carlossanlop
Assignees:	-
Labels:	blocking-clean-ci, area-VM-meta-mono, runtime-mono, os-tvos, Known Build Error
Milestone:	7.0.x

[kotlarmilos]

I don't see that happening anymore. If it starts happening again, I suggest we reopen the issue.