rrdom: an ad-hoc DOM for rrweb session data

[Yuyz0112]

For anyone familiar with rrweb's session data, a session has already contained enough information to do further analytic things. But to performant analysis to raw session data is not that simple.

For example, if we want to check 'whether the user has clicked a button that matches the CSS selector .btn-submit-form', we should:

1. replay the session
2. listen to the click event
3. when a click event was cast, check whether its target element matches .btn-submit-form

Since the replay deeply relies on the DOM API, we need a browser environment to do it, which is a bottleneck of performance when we need to do large scale analysis on many sessions.

So in this proposal, I suggest implementing an ad-hoc DOM called 'rrdom' as an environment to replay the session.

Why not jsdom?

jsdom is a feature-rich DOM implementation written in JS and can run in Node.js. Since jsdom wants to get close to the web standard, it's quite complex and has a lot of overhead for our usage.

What kinds of DOM API we should implement

We only need DOM manipulation APIs and CSS query selector APIs in rrdom.
To achieve this, we need a tree data structure to represent the DOM and provide the following public APIs:

• appendChild
• removeChild
• setAttribute
• querySelector
• querySelectorAll

How to do performant analysis with rrdom

The steps do not change, but the first step will be much faster since we will replay the session in rrdom.

None of the visual effects of replay will be present, but we can match CSS selectors when certain interactive events occurred.

Record a time-sensitive index

Sometimes we may want to build an index for the session. The index should contain some general information like:

1. clicked .btn-submit-form at 1500ms.
2. input at input[name="email"] at 3000ms.

This is useful if we want to build a UI that shows the events with the target descriptor(because rrweb only records an id of the target which is not user-friendly).

Also, it is useful if we want to some filter in the backend like 'show me the sessions contain click event on .btn-submit-form'.

The only thing we should be careful of is this index is time-sensitive. For example, if we clicked on the button element with id 1 at 1500ms and 3000ms, it may have a different class name or attributes with it.

Isomorphic or not

The first version of rrdom will be written in JS, so we can use it in both Node.JS and browser.

But it is also possible to implement rrdom in some language like rust. Which may provide better performance when we need to do large scale analysis.

[Yuyz0112]

@jay-khatri @Mark-Fenng

[Yuyz0112]

Reference: https://github.com/jsdom/js-symbol-tree

[YunFeng0817]

Can we directly use js-symbol-tree as our ad-hoc tree

[Yuyz0112]

js-symbol-tree looks solid. The only problem is it's using the symbol feature to avoid the metadata interfere original object.

In our scenario, we do not afraid of adding metadata to the original object. On the other hand, we need to consider the compatibility issue of symbols in browsers.

[YunFeng0817]

You are right , compatibility should be considered.
About the index, can we set the selector as key and elements with timeline as value?

[jay-khatri]

Hi there, thanks for the ping! Playing devils advocate here, but why do we need to replay this in a DOM environment? What's wrong with having something like:

• an onXEvent hook on record that spits out specific events (click on div.x.y at time z). This could be used by a backend for indexing.
• a generateXEvents on replay which spits out all the events for the current player.
  And each Xevent has an id so that when the replay is happening, we can point the user to the current event.

Are you saying that in order do something like generateXEvents, its easier to have rrdom? Just trying to understand.

[eoghanmurray]

because rrweb only records an id of the target which is not user-friendly

Another approach

Before discovering rrweb, I had a prototype built which recorded mutation and click events against the DOM using a CSS selector generated on the target as per https://github.com/antonmedv/finder

We could do something like that and record the CSS selector along with click/mutation events for static analysis.

(Ultimately this could replace the rrweb ID system, such that ID numbers don't need to be stored in the mirror, but I imagine that's another project 😄 )

[Yuyz0112]

@eoghanmurray unique CSS selector generation takes more cost on execution and storage.

[eoghanmurray]

I'm only imagining a small selection of events (click/mutation ... any more?) that the selector generation would have to be run?
And longer term if it could replace the storage of IDs that would be a huge storage saving.
But no worries I imagine it is a non-runner.

[jay-khatri]

@Yuyz0112 this could also be useful for doing more complex analysis of "Inactive" parts of the session, right? I.e. giving the user a visual cue as to what part of the session is inactive, etc.

[Yuyz0112]

@jay-khatri Yes it will be supported by this ad-hoc DOM.

[Yuyz0112]

@eoghanmurray

Since I've done some study on session storage optimization, are you facing some problem like the ID makes your full snapshot 'unstable'. So it's becoming hard for you to do some deduplication across sessions?

[eoghanmurray]

the ID makes your full snapshot 'unstable'

Yes, we strip out IDs from the JSON before storing them, as this allows us to partition the DOM and only store identical parts of the DOM once across many rrweb recordings. This results in reduction of about 90% in DOM storage requirements (it varys depending on the website). To recreate the mirror we rely on the fact that IDs are stored incrementally over a DOM tree walk — however this is not without it's bugs.