Add docs to better describe the .NET diagnostic experience

[noahfalk]

We've done lots of diagnostic work, but customers either aren't aware of it or don't know how it works. We need documentation on docs.microsoft.com to explain everything we've been building in a consistent and accessible way. Now that we have a landing page we need to fill it out much more. If the check list below is too much work for a single work item or hard to parcel out feel free to create additional items that track just one bullet and link them:

• Conceptual docs

  • Logging and Tracing
    • EventListener (it is mentioned but not described). Include init behavior.
    • Explain Activity IDs Document EventSource Activity ids #1347
    • DiagnosticSource and DiagnosticListener #3029
  • Symbol and Source Indexing
  • Diagnostic Port and Microsoft.Diagnostics.NetCore.Client

• Walkthroughs

  • Diagnosing CPU usage with dotnet-trace and VS (there is an existing example but it uses PerfView which is harder for many customers to work with)
  • Writing a custom EventSource, capture with dotnet-trace, view with VS #2902
  • E2E guide how to collect a dump and debug it with VS or LLDB #2903
  • Writing "fast" EventSource with WriteEventCore API "best practices". #2904
  • Diagnosing managed memory leaks with dotnet-gcdump. #2905
  • Diagnosing threadpool starvation #2907
  • Using EventSource's Start/Stop opcodes - cover best-practices and potential caveats. #2908
  • Counters "best practices" - when to use PollingCounter vs EventCounter, Incrementing vs non-Incrementing, thread safety, etc. #2909
  • Debugging dumps with cross-platform DAC. #2910
  • Guidance on using PerfView/other perf visualization tools. #2911
  • Guidance for library writers on creating an EventSource that is easily consumable. This includes adding EventCounters and what types of events are helpful for users. (based on conversation from Enhance lifetime monitoring in EventListener runtime#45247)

• EventSource-specific gaps: [Documentation] EventSource documentation improvement #1736

  • Increased detail on use cases of and differences between the various ctors on EventSource
  • Increased documentation on features such as custom keywords
  • Clarification on differences between consumption channels, i.e., nuget package, BCL, redist nuget package
  • Describe internal mappings of CLR types to eventing payload types, e.g., bool --> int of 0 or 1, DateTime --> FILETIME, etc. (per EventSource should mention some built-in types require special handling #2608)

• Reference

  • Known caveats to tools when used in prod environments. [Documentation] Document known caveats to diagnostic tools #1737
  • Add documentation to Diagnostic Client API

• Additional Work

  • Dotnet-gcdump collect causes OutOfMemoryException documentation Dotnet-gcdump collect causes OutOfMemoryException #2038
  • Correctly Document UNMANAGED_CODE_TIME vs CPU_TIME Correctly Document UNMANAGED_CODE_TIME vs CPU_TIME #1166
  • Retire duplicate documentation - Retire duplicate documentation on diagnostics repo #1682

[noahfalk]

cc @dotnet/dotnet-diag @shirhatti @davidfowl

[sdmaclea]

There is some work in flight.
dotnet/docs#13981
dotnet/docs#14110

[samsp-msft]

I'd also suggest:

• Whitepaper and sample agent for monitoring a containerized site/service
• Protocol docs for event pipe

[josalem]

IPC protocol is documented here: https://github.com/dotnet/diagnostics/blob/master/documentation/design-docs/ipc-protocol.md

It should probably be moved out of "design-docs" though, as it is an actual documentation of the protocol.

[noahfalk]

Whitepaper and sample agent for monitoring a containerized site/service

I think this would be very useful, but is probably a big enough scope of work that we treat it as its own feature that is scheduled during 5.0. The 3.0 goals were primarily to let people do ad-hoc analysis and investigation using these tools. Monitoring gets into other concerns such as alerting, triggered data collection, storage policies, data-transport and aggregation.