EventPipe events cannot always be read in realtime.

[wiktork]

Description

EventPipe events that are generated in a small batch with no subsequent events cannot be read in real-time by TraceEvent.

EventPipe events periodically get a timestamp. This timestamp informs TraceEvent that events that it has received so far can be ordered and dispatched. However, events after this timestamp are cached and are not dispatched until another timed event occurs.

This is described at here.

The TraceEvent logic that caches events that are not timestamped is at here.

@noahfalk @davmason

Reproduction Steps

Create an app that generates custom events similar to:

using System.Diagnostics.Tracing;

TestEventSource s = new TestEventSource();

while (true)
{
    Console.ReadLine();
  
    for (int i = 0; i < 10; i++)
    {
        s.TestEvent(i);
    }

}

[EventSource(Name = "TestProvider")]
class TestEventSource : EventSource
{
    public void TestEvent(int id) => WriteEvent(1, id);
}

Create an app that consumes these events in realtime:

Process[]? processes = null;
while (processes?.Length != 1)
{
    processes = Process.GetProcessesByName("EventPipeRepro");
    await Task.Delay(100);
}

int counter = 0;

DiagnosticsClient client = new DiagnosticsClient(processes.First().Id);
var session = client.StartEventPipeSession(new[]{
      new EventPipeProvider("TestProvider", System.Diagnostics.Tracing.EventLevel.Verbose, (long)TraceEventKeyword.All),
System.Diagnostics.Tracing.EventLevel.Verbose},
      false);

  
  
  var source = new EventPipeEventSource(session.EventStream);

  source.Dynamic.All += (e) =>
  {
      if (e.ProviderName == "TestProvider")
      {
          Console.WriteLine($"{counter++} {e.EventName} {e.PayloadString(0)}");
      }
      else
      {
          Console.WriteLine($"{e.ProviderName} {e.EventName}");
      }
  };
  source.Process();

Expected behavior

Every time events are generated the reader should be able to see all of them.

Actual behavior

The reader typically sees 1 event. The remaining events can only be seen once another run of events is generated.

Regression?

No

Known Workarounds

One of the events that is generated must be delayed so that it gets timestamped.

Configuration

Windows x64 .net 6 console app for both reader and writer. The issue is not configuration specific.

Other information

No response

[tommcdon]

@wiktork we are reviewing remaining work for 8.0 and have punted this one based off of remaining time. Please let us know if we should re-evaluate.