Rendering

The rendering process currently takes the following steps:

1. Recreate the state of the game as indicated in the raw replay data on a canvas.
2. Extract the canvas content as a webp image.
3. Use Whammy to create webm files from the webp images for each frame.

The current implementation:

1. renders several frames
2. reports progress
3. very small delay
4. repeat until replay is rendered

Problems

1. Rendering is very slow (it can take a few minutes for even a 30s replay at 1280x800 resolution and 60FPS)
2. While replays are being rendered the whole extension is locked up. If the menu is refreshed then the current rendered replay needs to be finished before the background page responds.
3. The finished webm file is huge.

Causes

1. Profiling shows that the majority of the time rendering is spent in the extraction of the canvas data to webp.
2. To prevent locking up the menu the rendering process is delegated to the background page. The canvas extraction takes place in the main thread of the background page which is also responsible for responding to replay info requests.
3. The webp extraction and Whammy compilation treats every frame as a key frame. We have no inter-frame compression. The resolution of replays is also usually very large compared to what I've seen mentioned elsewhere.

Solutions

Rendering performance

1. Parallelize the generation of frames.
2. Generate individual frames faster.

The current rendering process is extremely parallelizable. The generation of each frame (both the rendering to the canvas and the extraction only depend on the current frame). Web Workers currently do not have access to Canvas so the frames can't be generated in the workers directly. It may be possible to use a non-DOM canvas implementation like node-canvas within a web worker or OffScreenCanvas when it is available (see here for support updates and link to launch bug).

Individual frames can be generated faster by targeting something other than webp. Extracting the raw image data from the canvas seems promising when compared with webp extraction here. 50x faster. These could be shuffled to a web worker, converted to webp, and shuffled back. On the web worker side, it may be possible to initialize the passed data as ImageData which seems to be the format used within Chrome itself for the image conversion, but only the canvas seems to implement a native ImageData -> webp conversion. A project like webpjs could be used to do the conversion solely in JS. Alternatively a module in asm or WebAssembly could be used to the same effect. NPAPI has been deprecated/removed, but Native Client is one possible place to explore options for doing conversion. The discussion here looks at the possibility of using FFmpeg within NaCl. There are several examples online of doing image manipulation in web workers.

Here is some information on passing ImageData to web workers.

No matter what, it can be helpful when passing large amounts of data between the main page and a web worker to transfer ownership of the object. This removes the need to make a structured clone of the data every time something is passed back and forth. We can see an example here showing some performance possibilities.

The other issue is compilation of the image. If we want to stick with webm format then that still needs to be constructed from webp.

videoconverter.js (FFmpeg ported to JS) may be helpful, at least it has the ability to convert between several file types.

Another possibility is the MediaRecorder API which can generate a webm/mp4 directly from canvas. Current difficulties include:

1. MediaRecorder does not work on canvas on a non-visible page. (source?)
2. MediaRecorder/track does not seem to be able to take a specific time per frame if we already know it (which we do), so the output is going to be dependent on either the input fps or the live rendering of the replay to canvas. Maybe the solution here can be adapted to provide the frames at the correct times.
3. When recording the in-page-previewer with MediaRecorder, playback was slowed and the rendered file was also slow.

There are also a few issues with MediaRecorder in Chrome: can't capture on hidden elements and can't capture on backgrounded tabs. Some testing is needed to see if the second case also applies to background pages associated with extensions.