XUnitLogChecker: Frames are sometimes unresolved

[carlossanlop]

Examples from #93906:

• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-f0b140d21ce348bc89/System.IO.Compression.Brotli.Tests/1/console.7c296702.log?helixlogtype=result
• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-f1656fd4a7794c27a8/System.IO.Compression.Brotli.Tests/1/console.01a1cf6f.log?helixlogtype=result
• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-d61710e0f43545ab9d/System.IO.Compression.Brotli.Tests/1/console.795e54e9.log?helixlogtype=result

Printing the dumps is not very useful when they look like this:

0  Id: 1ea4.1430 Suspend: 0 Teb: 00894000 Unfrozen
ChildEBP RetAddr  
00b7da5c 76e44240 ntdll!NtWaitForMultipleObjects+0xc
00b7dbf0 7252f3c6 KERNELBASE!WaitForMultipleObjectsEx+0xf0
WARNING: Stack unwind information not available. Following frames may be wrong.
00b7dc74 7252f776 coreclr!GetCLRRuntimeHost+0x145706
00b7ddcc 7253637d coreclr!GetCLRRuntimeHost+0x145ab6
00b7de1c 7252f46b coreclr!GetCLRRuntimeHost+0x14c6bd
00b7de7c 725a7d9b coreclr!GetCLRRuntimeHost+0x1457ab
00b7df54 0b8f1533 coreclr!GetCLRRuntimeHost+0x1be0db
00b7df9c 0b80e2e5 0xb8f1533
00b7e074 0b801784 0xb80e2e5
00b7e15c 0a158358 0xb801784
00b7e240 0123bf82 0xa158358
00b7e27c 728500cb 0x123bf82
00b7e288 7257ddea coreclr!GetCLRRuntimeHost+0x46640b
00b7e6a4 7257df28 coreclr!GetCLRRuntimeHost+0x19412a
00b7e720 7257e5a5 coreclr!GetCLRRuntimeHost+0x194268
00b7e984 723708ed coreclr!GetCLRRuntimeHost+0x1948e5
00b7eaac 723704dc coreclr+0x408ed
00b7eae8 72370599 coreclr+0x404dc
00b7eb3c 723706c3 coreclr+0x40599
00b7eb9c 7236d30d coreclr+0x406c3
00b7eef0 723e8473 coreclr+0x3d30d
00b7f040 729857c2 coreclr+0xb8473
*** WARNING: Unable to verify checksum for hostpolicy.dll
00b7f0a0 72d3edc3 coreclr!coreclr_execute_assembly+0x92
00b7f0d0 72d7b9c9 hostpolicy+0xedc3
00b7f1f0 72d7b691 hostpolicy+0x4b9c9
00b7f22c 72d7cb3e hostpolicy+0x4b691
*** WARNING: Unable to verify checksum for hostfxr.dll
00b7f2d8 72f2425d hostpolicy!corehost_main+0x10e
00b7f3a4 72f2829e hostfxr!hostfxr_set_runtime_property_value+0x912d
00b7f410 72f26ec5 hostfxr!hostfxr_set_runtime_property_value+0xd16e
00b7f4a8 72f23eec hostfxr!hostfxr_set_runtime_property_value+0xbd95
00b7f56c 72f1ab58 hostfxr!hostfxr_set_runtime_property_value+0x8dbc
00b7f614 01363639 hostfxr!hostfxr_main_startupinfo+0x88
00b7f7dc 01363a9d dotnet+0x13639
00b7f830 013674e3 dotnet+0x13a9d
00b7f850 013673b7 dotnet+0x174e3
00b7f8ac 0136725d dotnet+0x173b7
00b7f8b4 01367548 dotnet+0x1725d
00b7f8bc 753a62c4 dotnet+0x17548
00b7f8d0 77e81ca9 kernel32!BaseThreadInitThunk+0x24
00b7f918 77e81c74 ntdll!__RtlUserThreadStart+0x2f
00b7f928 00000000 ntdll!_RtlUserThreadStart+0x1b
...

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Examples from #93906:

• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-f0b140d21ce348bc89/System.IO.Compression.Brotli.Tests/1/console.7c296702.log?helixlogtype=result
• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-f1656fd4a7794c27a8/System.IO.Compression.Brotli.Tests/1/console.01a1cf6f.log?helixlogtype=result
• https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-d61710e0f43545ab9d/System.IO.Compression.Brotli.Tests/1/console.795e54e9.log?helixlogtype=result

Printing the dumps is not very useful when they look like this:

0  Id: 1ea4.1430 Suspend: 0 Teb: 00894000 Unfrozen
ChildEBP RetAddr  
00b7da5c 76e44240 ntdll!NtWaitForMultipleObjects+0xc
00b7dbf0 7252f3c6 KERNELBASE!WaitForMultipleObjectsEx+0xf0
WARNING: Stack unwind information not available. Following frames may be wrong.
00b7dc74 7252f776 coreclr!GetCLRRuntimeHost+0x145706
00b7ddcc 7253637d coreclr!GetCLRRuntimeHost+0x145ab6
00b7de1c 7252f46b coreclr!GetCLRRuntimeHost+0x14c6bd
00b7de7c 725a7d9b coreclr!GetCLRRuntimeHost+0x1457ab
00b7df54 0b8f1533 coreclr!GetCLRRuntimeHost+0x1be0db
00b7df9c 0b80e2e5 0xb8f1533
00b7e074 0b801784 0xb80e2e5
00b7e15c 0a158358 0xb801784
00b7e240 0123bf82 0xa158358
00b7e27c 728500cb 0x123bf82
00b7e288 7257ddea coreclr!GetCLRRuntimeHost+0x46640b
00b7e6a4 7257df28 coreclr!GetCLRRuntimeHost+0x19412a
00b7e720 7257e5a5 coreclr!GetCLRRuntimeHost+0x194268
00b7e984 723708ed coreclr!GetCLRRuntimeHost+0x1948e5
00b7eaac 723704dc coreclr+0x408ed
00b7eae8 72370599 coreclr+0x404dc
00b7eb3c 723706c3 coreclr+0x40599
00b7eb9c 7236d30d coreclr+0x406c3
00b7eef0 723e8473 coreclr+0x3d30d
00b7f040 729857c2 coreclr+0xb8473
*** WARNING: Unable to verify checksum for hostpolicy.dll
00b7f0a0 72d3edc3 coreclr!coreclr_execute_assembly+0x92
00b7f0d0 72d7b9c9 hostpolicy+0xedc3
00b7f1f0 72d7b691 hostpolicy+0x4b9c9
00b7f22c 72d7cb3e hostpolicy+0x4b691
*** WARNING: Unable to verify checksum for hostfxr.dll
00b7f2d8 72f2425d hostpolicy!corehost_main+0x10e
00b7f3a4 72f2829e hostfxr!hostfxr_set_runtime_property_value+0x912d
00b7f410 72f26ec5 hostfxr!hostfxr_set_runtime_property_value+0xd16e
00b7f4a8 72f23eec hostfxr!hostfxr_set_runtime_property_value+0xbd95
00b7f56c 72f1ab58 hostfxr!hostfxr_set_runtime_property_value+0x8dbc
00b7f614 01363639 hostfxr!hostfxr_main_startupinfo+0x88
00b7f7dc 01363a9d dotnet+0x13639
00b7f830 013674e3 dotnet+0x13a9d
00b7f850 013673b7 dotnet+0x174e3
00b7f8ac 0136725d dotnet+0x173b7
00b7f8b4 01367548 dotnet+0x1725d
00b7f8bc 753a62c4 dotnet+0x17548
00b7f8d0 77e81ca9 kernel32!BaseThreadInitThunk+0x24
00b7f918 77e81c74 ntdll!__RtlUserThreadStart+0x2f
00b7f928 00000000 ntdll!_RtlUserThreadStart+0x1b
...

Author:	carlossanlop
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[carlossanlop]

Another example: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-93906-merge-73d789b084234d60b0/System.IO.Compression.Brotli.Tests/1/console.68d2b2ea.log?helixlogtype=result

Thread Id: 0x1e
      Child SP               IP Call Site
 0xffa24af9d810 0xffa24a9e22ec libclrjit.so!?? at ??:0:0
 0xffa24af9d890 0xffa24a9d0408 libclrjit.so!?? at ??:0:0
 0xffa24af9d8e0 0xffa24a9d6874 libclrjit.so!?? at ??:0:0
 0xffa24af9d990 0xffa24a9d5430 libclrjit.so!?? at ??:0:0
 0xffa24af9da70 0xffa24a9d7f64 libclrjit.so!?? at ??:0:0
 0xffa24af9e0c0 0xffa24a9e1910 libclrjit.so!?? at ??:0:0
 0xffa24af9e180 0xffe3222f56e0 libcoreclr.so!?? at ??:0:0
 0xffa24af9e1f0 0xffe3222f586c libcoreclr.so!?? at ??:0:0
 0xffa24af9e260 0xffe3222f67a0 libcoreclr.so!?? at ??:0:0
 0xffa24af9eef0 0xffe3223453c8 libcoreclr.so!?? at ??:0:0
 0xffa24af9eff0 0xffe322344bf8 libcoreclr.so!?? at ??:0:0
 0xffa24af9f120 0xffe322344220 libcoreclr.so!?? at ??:0:0
 0xffa24af9f1f0 0xffe3223438c0 libcoreclr.so!?? at ??:0:0
 0xffa24af9f2c0 0xffe322387814 libcoreclr.so!?? at ??:0:0
 0xffa24af9f3e0 0xffe322386a28 libcoreclr.so!?? at ??:0:0
 0xffa24af9f500 0xffe322385dc0 libcoreclr.so!?? at ??:0:0
 0xffa24af9f580 0xffe322385bf8 libcoreclr.so!?? at ??:0:0
 0xffa24af9f5c0 0xffe322380ac0 libcoreclr.so!?? at ??:0:0
 0xffa24af9f740 0xffe322381040 libcoreclr.so!?? at ??:0:0
 0xffa24af9f770 0xffe322385b40 libcoreclr.so!?? at ??:0:0
 0xffa24af9f7b0 0xffe3227ff58c libcoreclr.so!?? at ??:0:0
 0xffa24af9f860 0xffe322c93ce0 ld-musl-aarch64.so.1!
----------------------------------

[hoyosjs]

This looks like lack of symbols on the test machine (for some of these) and then missing function table data in the dump (this is a known issue and the only workaround is to modify the registry).

[JulieLeeMSFT]

If it is a missing symbol, we need to find out why it is missing.

[hoyosjs]

@carlossanlop

I haven't yet found newer examples on linux, but for windows there's two issues.

• Nothing installs sos so you'll never get managed stacks. See how runtime tests do it:

  runtime/src/tests/Common/helixpublishwitharcade.proj

  Lines 783 to 787 in 777f9d9

  	<HelixCorrelationPayload Condition="'$(TestWrapperTargetsWindows)' == 'true'" Include="dotnet-sos">
  	<Destination>sos</Destination>
  	<Uri>https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/flat2/dotnet-sos/$(DotnetSosVersion)/dotnet-sos.$(DotnetSosVersion).nupkg</Uri>
  	</HelixCorrelationPayload>

  and then the call to dotnet sos install as part of the workitems command.
• Nothing sets a symbol path. You need to set _NT_SYMBOL_PATH to point to the folder that has runtime symbols. Like this (with the correct payload path):

set "_NT_SYMBOL_PATH=E:\Temp\libs-crash\correlation-payload;E:\Temp\libs-crash\correlation-payload\shared\Microsoft.NETCore.App\9.0.0"

[JulieLeeMSFT]

Reassigning to @carlossanlop.

[ericstj]

Looks like we missed some additional setup steps required for the xunit symbolizer tool. @hoyosjs is there anything to audit here to make sure we copied all the relevant steps from the runtime test deployment into libs?