updatePixels() can't be called in WEBGL Mode

[aferriss]

Most appropriate sub-area of p5.js?

• Accessibility (Web Accessibility)
• Build tools and processes
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Friendly error system
• Image
• IO (Input/Output)
• Localization
• Math
• Unit Testing
• Typography
• Utilities
• WebGL
• Other (specify if possible)

Details about the bug:

When using webGL mode, I'm not able to call updatePixels(). An error is given that updatePixels isn't a method on the current renderer. This applies to instances created with createGraphics in webGL mode as well.

Here's a sample sketch showing the issue.
https://editor.p5js.org/aferriss/sketches/LlY7q2OLE

• p5.js version: 1.3.1
• Web browser and version: Chrome
• Operating System: Max OSX
• Steps to reproduce this:

1. Create a sketch in webgl mode createCanvas(400, 400, WEBGL)
2. Load some pixels and do some pixel array manipualation
3. Call updatePixels()

[stalgiag]

It somehow seems that this just never existed which is extremely odd. It seems as though you are the first person to try to do pixel manipulation of a WebGL canvas and make an issue about how it doesn't work. So strange.

Anyways, we should likely add this or at least add a stub function so this doesn't crash sketches.

I am not sure what the best approach would be for setting the pixels in the WebGL context. I don't think there is a direct way to write pixels so some process of conversion would have to happen.

[aferriss]

Yeah, I thought it was weird too, but maybe it's kind of an unusual thing to be doing 😅

I think in terms of implementation, all we would need to do is essentially just create a texture using the current pixel data and run it through a shader that just draws that texture to the screen.

It's definitely going to be a slow operation, but good for a case when you want to go shader -> pixels and back again.

In my case I worked around it by just creating another non-webgl graphics object, drawing my webgl texture to there and then doing my pixel operations.

[aferriss]

Here's a small example of how it could be implemented just using existing functions. I'm not sure it's the most efficient but it works 🤷

https://editor.p5js.org/aferriss/sketches/5YQevK79X

[fedeb95]

I was trying to do this too and it didn't work... would be cool to set directly pixels using WEBGL context

[aferriss]

I think you may be better off just drawing your webGL buffer to a graphics object that is not in webGL mode, doing your pixel manipulation there, and then drawing that back to the webGL canvas.
Here's a small example. It's sort of hacky, but maybe works better than what I suggested above.

https://editor.p5js.org/aferriss/sketches/BEGhjqE6-

[JuanIrache]

Any updates on this? Thank you.

[Tezumie]

I was having this same issue, I use pixels to create a filter for my sketches. doesn't work in WebGL :(

Any updates?

[Real-John-Cheung]

Seems that it's still a problem in 2023, any news?

[davepagurek]

It's on the roadmap now, so it won't get lost! (You can see how we've sorted WebGL issues here: https://github.com/orgs/processing/projects/5)

There's a lot of WebGL things in there so this one isn't an immediate priority. However, if anyone is interested in tackling this in the mean time, feel free to let me know, and I can help guide the process!

[nikhilhvr]

Hey, @davepagurek I want to work on this issue. Do you have any suggestions that might help me regarding this issue?

[davepagurek]

Thanks @nikhilhvr, I'll assign this to you!

I think @aferriss's comment #5155 (comment) is probably the best approach for now -- when one wants to updatePixels, we put those pixels onto a p5.Image, and then draw that image to the canvas. Some little nuances to consider:

• p5.Image doesn't have a concept of pixel density, but graphics/canvases do, so we'll need to take care that the image has the right size to capture all the pixels
• There might be transforms, a camera, and an image mode that would affect how the image gets drawn to the canvas. We'd probably need to put our code in a push()/pop() and reset all those things to make sure the image we draw takes up the full canvas.
• We might want to keep the image around so that we aren't creating a new one each time the renderer tries to update pixels. Maybe we should do something similar to the internal canvas of p5.MediaElement where we create it if it doesn't exist, and resize it when we're about to use it if it's not the right size:

  p5.js/src/dom/dom.js

  Lines 3143 to 3159 in 9ee0fa7

  	if (!this.canvas) {
  	this.canvas = document.createElement('canvas');
  	this.drawingContext = this.canvas.getContext('2d');
  	this.setModified(true);
  	}
  	// Don't update the canvas again if we have already updated the canvas with
  	// the current frame
  	const needsRedraw = this._frameOnCanvas !== this._pInst.frameCount;
  	if (this.loadedmetadata && needsRedraw) {
  	// wait for metadata for w/h
  	if (this.canvas.width !== this.elt.width) {
  	this.canvas.width = this.elt.width;
  	this.canvas.height = this.elt.height;
  	this.width = this.canvas.width;
  	this.height = this.canvas.height;
  	}