[p5.js 2.0 Bug Report]: (Performance) Drawing video elements in WebGL copies the canvas unnecessarily

[davepagurek]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

p5.js version

2.2.2

Web browser and version

Chrome

Operating system

Windows

Steps to reproduce this

Steps:

1. Create a video element
2. Create a WebGL canvas
3. Draw the video to the canvas

This isn't super noticeable on Mac, but on Windows (older hardware especially), the frame rate of the video being drawn to the WebGL canvas is a lot lower than the frame rate of the video element.

This seems to be happening because we call ensureCanvas(), which draws the element to a canvas element, but then we don't actually use that canvas element for the texture later:

p5.js/src/webgl/p5.Texture.js

Lines 106 to 109 in 5b74aa6

	if (this.src._ensureCanvas) {
	this.src._ensureCanvas();
	}
	textureData = this.src.elt;

So for video elements, this is extra work that goes unused. I need to investigate more what's going on for webcam feeds, though, where ensureCanvas additionally mirrors the video feed if you ask for it. That may be broken as a feature since we ignore the canvas currently when getting the texture.

Note that I think the fix is not to use the canvas as a texture instead of the original element, because that extra copy is pretty heavy on older hardware, and it'd be great to be able to remove it.

Snippet:

Live: https://editor.p5js.org/davepagurek/sketches/9nJobnGc-

Try toggling on and off testWebGL to see the difference between looking at the video element directly vs showing it used as a texture on a WebGL canvas.

let vid
let testWebGL = true
function setup() {
  createCanvas(720, 405, WEBGL);
  vid = createVideo('https://upload.wikimedia.org/wikipedia/commons/transcoded/4/49/Nature_montage_around_Aberfeldy.webm/Nature_montage_around_Aberfeldy.webm.1080p.vp9.webm')
  if (testWebGL) vid.hide()
  vid.volume(0)
  vid.size(width, height)
  vid.loop()
}

function draw() {
  background(220);
  if (testWebGL) {
    imageMode(CENTER)
    image(vid, 0, 0, width, height)
  }
}

[davepagurek]

Update: createCapture(VIDEO, { flipped: true }) is also broken :') I'll fix that too.