add a method `fillImageData` in CanvasImageData

[Zhang-Junzhi]

For now, the only possible way to fetch the image data in a canvas is to call getImageData, which returns a newly created ImageData.

However, sometimes the buffer where we want to store the image data has already been made, so it's useful to fill the data into the exisitng buffer, rather than creating a new one, to avoid data copying, so we can get better performance.

This is especially useful in tandem with WebAssembly.Memory. For example, let's consider the following pesudo code:

//Step 1: Allocate memory for wasm
const memory = new WebAssembly.Memory(...);
const imageData = new ImageData(new Uint8ClampedArray(memory.buffer), ...);

//Step 2: Do some work in canvas, and get the resulted image data for future use in wasm
offscreenCanvasContext.drawImage(...);
offscreenCanvasContext.fillImageData(imageData, sx, sy, sw, sh); //If sw, sh are beyond the room of the data buffer, an exception is thrown.

Without a method like fillImageData, we have to first call getImageData to create a new ImageData, and then copy the data in ImageData to the wasm memory, which results in an unnecessary performance cost.

So I propose to add a new method fillImageData in CanvasImageData.

[domenic]

This makes sense to me. Any canvas implementers interested? /cc @whatwg/canvas

[nical]

While it makes sense in the context of this wasm example, I want to note that since reading back memory from the GPU to the CPU is very expensive, an API such as this one would only be a potential improvement if the buffer really needs to be used on the CPU (and not turned back into a graphics object of some sort).
Also, since this operation would be very expensive for GPU canvas implementations, it would make a lot of sense to make it asynchronous. It's not immediately obvious to me how an async version of that would look like, though.

[fserb]

It seems to me that the biggest time spent doing getImageDate is not the copy or creation of the buffer. So it feels like we would be adding a new entry point for performance that doesn't address 90% of the performance problem, which may limit a lot the practical use cases of an api like this.

[dsanders11]

I came here to file this same issue, as I also think it's an important performance improvement. WebGLRenderingContext has readPixels which provides a way to fill an existing ImageData, but using WebGL for 2D stuff is overkill and not trivial for those (such as myself) unfamiliar with WebGL.

I was going to suggest an existing ImageData could be an optional final arg to getImageData, but fillImageData seems reasonable as well, especially if the API needs to be async.

Perhaps I can provide a more compelling use case. A standard way of getting and using webcam frames is to capture the webcam to a video element, then use drawImage to draw the frame to a canvas, then use getImageData to get the ImageData object, at which point pixel manipulation can finally be done. If you wanted to add a watermark to video for WebRTC, AFAIK this is the only way to do it. You'd add the watermark then use putImageData to put it back to a canvas and then you can use that as the video source.

I have a use-case where I use video from the webcam for barcode scanning using this technique of getting ImageData for a webcam frame. It works, but there is a quite a bit of thrashing going on with memory usage due to the fact that trying to keep up with 25 FPS means for a 640x480 video capture you're creating an ~1.2 MB ImageData (4 bytes per pixel with RGBA) every frame, totaling 30 MB/sec. This amount of large garbage doesn't play well with the garbage collection. Using the task manager in Chrome I can see that the memory usage for the tab fluctuates quite a bit. This has caused continual problems with lower memory devices, or devices with a high memory usage.

Basically any use-case where you want to use or manipulate webcam data suffers from the performance impact of allocating a new ImageData for every frame. Here is a CodePen which simply uses the technique I mentioned above to draw video from a webcam to a canvas, leaving out any actual usage of the ImageData (say, for finding and reading a barcode). You can change the variables at the top of the JS to get a finer-grained view of performance. On Chrome 70, I'm seeing memory usage for the tab fluctuating between 115 MB - 190 MB. If you increase the image size to 1280x960, memory usage stays in that same range, but CPU usage for the tab doubles, due to the higher garbage collection workload. If you set the FPS to 1, CPU usage is much lower, and you can watch the memory usage for the tab slowly climb each second until it hits the garbage collection threshold and drops back down. With the proposed fillImageData, you'd go from allocating and garbage collecting 30 MB/sec (for 640x480@25 FPS) to a static ~1.2 MB allocation which is reused and requires no garbage collection.

@nical, @fserb, @domenic, any thoughts on this more detailed write-up for a real-world use-case affected by this performance impact? The change would minimize the amount of garbage which otherwise gets created, which is where the performance improvements really come in.