dotnet-gcdump and perfview's gcdump do not work on 5.0 runtimes

[davmason]

If you collect a gc dump with either dotnet-gcump or perfview's gcdump command from a live process you will see output like this:

Name
UNKNOWN 0x7ffabf7e46a0
UNKNOWN 0x7ffabf811688 (Bytes > 10K)
UNKNOWN 0x7ffabf7a7a48 (Bytes > 10K)
UNKNOWN 0x7ffabf7ecb50
UNKNOWN 0x7ffabf6e6768 (Bytes > 1K)
[static vars]
UNKNOWN 0x7ffabf7ecbe0 (Bytes > 1K)
UNKNOWN 0x7ffabf7e46a0 (Bytes > 1K)
[strong Handles]
UNKNOWN 0x7ffabf7a2990 (Bytes > 1K)
UNKNOWN 0x7ffabf7a7a48
UNKNOWN 0x7ffabf806c20
UNKNOWN 0x7ffabf6e6768
UNKNOWN 0x7ffabf814298
UNKNOWN 0x7ffabf7b1c80
UNKNOWN 0x7ffabf7a2990
UNKNOWN 0x7ffabf7aeb78
UNKNOWN 0x7ffabf812218
[local vars]
UNKNOWN 0x7ffabf814cb0
UNKNOWN 0x7ffabf7c0d28
[pinning Handles]
[static var Type(0x7ffabf814cb0).s_in]
[static var Type(0x7ffabf812218).s_out]
[static var Type(0x7ffabf7a2990).s_table]
[other roots]
[.NET Roots]
[static var Type(0x7ffabf7aeb78).s_dataStore]
ROOT

It works on 3.1 but fails on 5.0 runtimes. Since both tools are failing I suspect it is a runtime issue where we're not emitting the right events.

Tagging subscribers to this area: @Maoni0
Notify danmosemsft if you want to be subscribed.

Tagging subscribers to this area: @tommcdon
Notify danmosemsft if you want to be subscribed.

[josalem]

uh oh! Is it possible this can be tied back to the changes for the new Pinned Object Heap? GCHeapDumps work via callbacks while the GC walks the heap. If the events changed or format of the type info changed, it is quite possible the GCHeapDump event parsing logic is failing to map type info to events in the stream. I don't have immediate bandwidth to triage this, but I'll take a look hopefully later this week.

CC @brianrob @VSadov

[VSadov]

It is possible it is caused by Pinned Heap, although I do not recall if events changed for this.

[davmason]

I looked a little bit and it seems that BulkType events are not being fired. Here is a log from perfview, BulkTypeEventCount:0 and BulkNodeEventCount:1 seems very suspicious.

*****  Attempting a ETW based DotNet Heap Dump.
  0.0s: Requesting a .NET Heap Dump
Found Module System.Private.CoreLib.dll ID 0x7ffaa4694020
Found Module sampleapp.dll ID 0x7ffaa4aa32c8
Found Module system.runtime.dll ID 0x7ffaa4aa4e58
Found Module system.console.dll ID 0x7ffaa4b35030
Found Module system.threading.dll ID 0x7ffaa4b3d0f0
Found Module system.text.encoding.extensions.dll ID 0x7ffaa4b62650
  3.1s: Dot Net Dump Started...
Found a Gen2 Induced non-background GC Start at 266.522 msec GC Count 12
  3.1s: Making  GC Heap Progress...
  3.1s: DotNet GC Complete.
Found a GC Stop at 288.317 for GC 12, ignoring events from now on.
Processing Heap Data, BulkTypeEventCount:0  BulkNodeEventCount:1  BulkEdgeEventCount:1
Error: Did not have a type definition for typeID 0x7ffaa48ba6f8
Error: Did not have a type definition for typeID 0x7ffaa48eaa40
Error: Did not have a type definition for typeID 0x7ffaa48eaf48
Error: Did not have a type definition for typeID 0x7ffaa48eb198
Error: Did not have a type definition for typeID 0x7ffaa48eb3e8
Error: Did not have a type definition for typeID 0x7ffaa48eb600
Error: Did not have a type definition for typeID 0x7ffaa4790ed0
Error: Did not have a type definition for typeID 0x7ffaa48e4868
Error: Did not have a type definition for typeID 0x7ffaa493f5f0
Error: Did not have a type definition for typeID 0x7ffaa4966cb0
Error: Did not have a type definition for typeID 0x7ffaa479a498
Error: Did not have a type definition for typeID 0x7ffaa4967ef0
Error: Did not have a type definition for typeID 0x7ffaa48b99f8
Error: Did not have a type definition for typeID 0x7ffaa48b9b60
Error: Did not have a type definition for typeID 0x7ffaa4aa38d8
Error: Did not have a type definition for typeID 0x7ffaa4aab588
Error: Did not have a type definition for typeID 0x7ffaa4aabb30
Error: Did not have a type definition for typeID 0x7ffaa4aac1d0
Error: Did not have a type definition for typeID 0x7ffaa4aaab58
Error: Did not have a type definition for typeID 0x7ffaa4aad990
Error: Did not have a type definition for typeID 0x7ffaa4ae4fc0
Error: Did not have a type definition for typeID 0x7ffaa4ae56f0
Error: Did not have a type definition for typeID 0x7ffaa4ae5d18
Error: Did not have a type definition for typeID 0x7ffaa4aea068
Error: Did not have a type definition for typeID 0x7ffaa4aad548
Error: Did not have a type definition for typeID 0x7ffaa4aeff08
Error: Did not have a type definition for typeID 0x7ffaa4b008b8
Error: Did not have a type definition for typeID 0x7ffaa4b002c8
Error: Did not have a type definition for typeID 0x7ffaa4b0c488
Error: Did not have a type definition for typeID 0x7ffaa4b08b10
Error: Did not have a type definition for typeID 0x7ffaa4b0c568
Error: Did not have a type definition for typeID 0x7ffaa4b0d268
Error: Did not have a type definition for typeID 0x7ffaa4b07208
Error: Did not have a type definition for typeID 0x7ffaa479ea08
Error: Did not have a type definition for typeID 0x7ffaa4b347a8
Error: Did not have a type definition for typeID 0x7ffaa4b34df0
Error: Did not have a type definition for typeID 0x7ffaa4914ae8
Error: Did not have a type definition for typeID 0x7ffaa4b62240
Error: Did not have a type definition for typeID 0x7ffaa4b640c8
Error: Did not have a type definition for typeID 0x7ffaa4b65fe8
Error: Did not have a type definition for typeID 0x7ffaa4b692b0
Error: Did not have a type definition for typeID 0x7ffaa4b68878
Error: Did not have a type definition for typeID 0x7ffaa4b91418
Error: Did not have a type definition for typeID 0x7ffaa4b93570
Error: Did not have a type definition for typeID 0x7ffaa4b6ecc0
Error: Did not have a type definition for typeID 0x7ffaa4b93f38
Error: Did not have a type definition for typeID 0x7ffaa4b94238
Error: Did not have a type definition for typeID 0x7ffaa4b97768
Error: Did not have a type definition for typeID 0x7ffaa4b97a30
Error: Did not have a type definition for typeID 0x7ffaa4b9e098
Error: Did not have a type definition for typeID 0x7ffaa4b9dfc8
Error: Did not have a type definition for typeID 0x7ffaa491a020
Error: Did not have a type definition for typeID 0x7ffaa4b9e290
Error: Did not have a type definition for typeID 0x7ffaa4b9efa0
Error: Did not have a type definition for typeID 0x7ffaa4bb05c8
Error: Did not have a type definition for typeID 0x7ffaa4bb1630
Error: Did not have a type definition for typeID 0x7ffaa4bb31b0
Error: Did not have a type definition for typeID 0x7ffaa4b346c8
Error: Did not have a type definition for typeID 0x7ffaa4b69dc0
Error: Did not have a type definition for typeID 0x7ffaa4bb5290
Error: Did not have a type definition for typeID 0x7ffaa4bb6380
Error: Did not have a type definition for typeID 0x7ffaa4bb6888
  3.2s: Shutting down ETW session
  3.5s: ETW Listener dieing
  3.5s: Waiting for shutdown to complete.
[  3.5s: Done Dumping .NET heap success=True]

[josalem]

Hmm that looks like the behavior from 3.0 where the type system wasn't being flushed so the events were never firing. dotnet/coreclr#26270 fixed that in the 3.1 timeframe. Next week I'll take a look under a debugger and see if I can find anything.