Migrate to push-based data flow for tracer

[aryanjassal]

Specification

Currently, the tracer is pull-based. The events are generated and added to a global queue, which a generator can consume. This is bad, as if no consumers are present, then the events can start building up indefinitely.

To fix this, we need to adopt a push-based dataflow, where there is a source of events which are all consumed in real-time by subscribers via callbacks. This approach draws inspiration from how rxjs handles this.

For this implementation, however, rxjs is unnecessary complexity and we can make do with only WebStreams or EventEmitters internally. I'm leaning towards using streams, as streams handle backpressure for us if the consuming callbacks are slower than incoming data. However, that might mean an observable slowdown of the program if the backpressure buffer is full. To avoid this, we may need to create an unbound buffer, but that would again result in the same issue where the buffer can grow indefinitely. It would be a bit more manageable here, as events can be ignored if no consumers have been registered. This would need some discussion.

Additional context

Tasks

1. Switch from pull-based to push-based design
2. Incorporate rxjs or streams to manage resources
3. Should support multiple consumers or subscribers
4. Add benchmarks for different approaches like observables and webstreams

[linear]

ENG-616

[aryanjassal]

@tegefaulkes @CMCDragonkai need some inputs for this

[CMCDragonkai]

I originally wanted to integrate rxjs observables into Polykey library. But otherwise libraries usually expose event emission that can be turned into an observable flow. So you should be able to do both. Keep the libraries doing event emission as normal, then use the wrapper functions provided by rxjs and ixjs to then turn it into observables. Idea is observable subjects becomes part of the Polykey core.

[aryanjassal]

Okay, so I need to basically create an Event emitter here (not streams, that could be a wrapper if we need it) and use an unbound queue to let the events grow as needed. If we allow multiple callbacks, then this introduces another interesting point. If we queue stuff in the memory for multiple callbacks and one callback is slower than others, that would hold up other callbacks before that element can be removed from the queue. On the other hand, we can create queues per callback or event handler, but that would result in a massive increase in the required memory.

Also, we probably don't need to use AbstractEvents but I'm asking to confirm this. Do we want to use base events or use abstract events?

[CMCDragonkai]

No we are using js-events as the foundation.

[CMCDragonkai]

This requires empirical observation. No need to theory craft.

[tegefaulkes]

We'll need to use Benny to do benchmarks of the different methods. The main problem here is that tracing could end up blocking the main process in a crippling way. So the way we inplement matters.

We need to explore the usage of webstreams, observables and eventTarget to see how these methods of data-flow perform. We also need to explore how much buffering will help here.

Compare each method to

1. writing to a open file descriptor.
2. writing to stdout.
3. writing to console.log.
4. just dropping the data in memory.

Really we could implement a stream and observable method at the same time. they each have their pros and cons. I'd prefer a webstream for piping into a file for the back-pressure support. While an observable would be more useful for debugging in real-time.

[aryanjassal]

I went over this with Brian, and this is what we came up with.

Firstly, we need to create an observables foundation. Brian walked me through the basics of rxjs observables for this. As we know, observables are completely synchronous, which means they block the main loop execution to ensure order. This goes against tracer being transparent, as the very nature of observables will cause the main process to take a hit.

The approach we finalised is basically using observables as a potential egress point, but also registering a webstream to handle the events. As webstreams internally buffer the events, we get a free unbound buffer which adapts to the pace at which the consumer is consuming data.

In conclusion, first I will integrate observables into the codebase, then add a webstream wrapper around it to get a quick-and-easy buffer over the incoming data. I will write benchmarks to back the claims with empirical data.

[CMCDragonkai]

Observables are synchronous because "push" is synchronous. It's basically a function 1 calling function 2 calling function 3. This is fine - because the amount of work taking place isn't that much. By async sink, I'm talking about the final sink point, if that ends up calling an async IO operation, that then goes into the event loop and it is async.

[CMCDragonkai]

I'm not sure if webstreams make sense here? The idea behind observables are purely object oriented. You only use web streams when it becomes IO. I would not make these sorts of things a web stream unless it's doing IPC. An async file sink is not exactly IPC and therefore doesn't need to be a webstream.

[aryanjassal]

Eventually, we will need to use spans as decorators. We can handle doing that kind of like how js-contexts handles it. We can have decorators when an object needs to be traced, and also support inline higher-order functions when decorators are not viable anymore.

When we create a class and we want it to be traced, it needs its parent context to know where we are being traced inside — what is our parent. This is generally not possible as the callee will not know who the caller was generally. However, it becomes possible in meta programmable languages.

Javascript supported it before, where callee could access the caller object via the following approach. This, however, is now deprecated and not supported under strict mode.

function method() {
  const caller = method.caller;
}

But we have already solved this problem in js-contexts. We can extend it by adding another field for spanId, so the context object will look like this, explicitly tracking the span. If spanId is unset, we can assume the span belongs to the root.

{
  timer,
  signal,
  spanId,
}

Note that after talking with Roger about this, the conclusion we arrived at is to not worry about where the span ended, only that it actually ended — leaks will be found by observing open spans, not which context they ended up in before being closed.

[CMCDragonkai]

There should be 2 visualisations - one a vertical lines and forking system - and another a dynamic graph that shows the concurrent trace lines. Ideally they should be correlated visualisations over the time domain.