Support for drawing to/reading from framebuffers

[davepagurek]

Increasing Access

createGraphics is currently the recommended way to create a texture and use it elsewhere in a sketch. While this is very flexible and approachable, it unfortunately comes with inherent performance implications (#5128) that are currently are hard to work around within p5. Framebuffers are the native WebGL solution to this task with the best potential performance.

Giving users access to better performance with, ideally, a simple and familiar interface would let their sketches take better advantage of hardware capabilities and give viewers with lower-end hardware a much smoother viewing experience.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature request details

While for technical reasons it will not be feasible to make all graphics used as textures use Framebuffers under the hood, we believe creating an interface for Framebuffers that fits well with existing p5 features will still be beneficial.

Inspiration

• p5.Framebuffer
• p5Fbo
• openFrameworks ofFbo

High level features

• A Framebuffer class can be created with createFramebuffer()
• It may only be used in WebGL environments
• If WebGL2 is enabled, antialiasing is supported
  • We have decided, at first at least, we will not try to provide an antialiasing fallback on WebGL1
  • The most practical fallback option would be to render at 2x pixel density. However, this has a noticeable performance impact which is especially large on the sorts of older devices that don't have WebGL2 support. For that reason, we believe a more reasonable option is to degrade gracefully to a 1x density non-antialiased Framebuffer if only WebGL1 is available.
• The Framebuffer can be passed into texture() or setUniform() or image() like a graphic can

Implementation details

• Things that aren't part of the main motivation but we would get as a side effect:
  • Writing in floating point format
    • This has been asked for directly, but also indirectly after people notice artifacts in feedback sketches
    • Readable depth information (not essential for v1)
    • p5.Framebuffer allows this to be read via the .depth property of a framebuffer
• How does one choose a framebuffer to draw to?
  • Prior art:
    • p5.Framebuffer: fbo.draw(() => /* ... */)
    • p5Fbo, ofFbo: fbo.begin(); /* ... */ fbo.end();
  • Probably the begin/end version is more similar to push/pop and is what we should go with
    • The callback function version could also potentially be added on top of this as a way to avoid accidental improper nesting of begin/end, but this is not essential
• Different size support
  • Each fbo has to come with its own camera in order to be able to properly render to a size different from the main canvas
• We probably do not want to (at least initially) support all texture formats
  • By making users ask for either UNSIGNED_BYTE (default) or FLOAT, this lets us pick the appropriate mode for the environment (WebGL1, WebGL2, etc)
  • Antialiasing support will need lots of testing (p5.Framebuffer supports this but runs into cross-browser version issues often) so letting us pick texture modes that work will give us an avenue for resolving these sorts of issues

[aferriss]

Thanks for making this writeup Dave, very exciting! I have a couple quick thoughts:

1. Would this change also require us to add a friendlier api for creating and using p5.Texture objects? Or would you just pass around the framebuffer itself?
2. Should we also add support for the HALF_FLOAT data type? I believe it is more widely supported than FLOAT on mobile hardware
3. I think we should consider creating an api for reading a framebuffer back to pixels.
4. Can we outline some of the accessibility arguments that we talked about yesterday re: performance?
5. Can framebuffers also be created for p5.Graphics (probably yes, but does this get confusing)?
6. Is framebuffer the best name to use here? I know that's the technical term, but just want to double check if there is anything else less graphics jargon-y that could help newcomers find this feature.
7. Should a framebuffer try to emulate the previously used camera, or just always construct itself with a default?
8. Should we allow users some way to set the texture min / mag / wrap settings when constructing the framebuffer (I know I find this useful).

[davepagurek]

1. I think we could just pass in the framebuffer itself, similar to a p5.Graphics, and optionally provide texture accessors on the framebuffer for its color texture and depth texture like in p5.Framebuffer if people want to access any additional textures it might have.
2. HALF_FLOAT makes sense. Do you think it should be a separate option people should pick or something we try to fall back to if FLOAT isn't available?
3. Makes sense, I think we can support the same pixel reading as p5.Graphics.
4. In addition to what's in the issue description about reducing the hardware requirements to run p5 sketches (especially useful for students working on home computers instead of school computers), there is currently a fairly large gap between writing a sketch in a way that is easy to understand and a way that will run well for as many users as possible. This would help bridge that gap by offering a more incremental step from the p5 people are used to and more powerful but complicated and industry-focused tools.
5. I agree, I think they should exist, but we should have some clear examples showing that in most cases you can use a WebGL main canvas + framebuffers instead of using any graphics. I think the only reason to use framebuffers with graphics would be if you have a 2D main canvas that you want to embed some WebGL content in, and need an extra WebGL layer inside that graphic.
6. I'm open to a friendlier name, do you have any suggestions? It's kind of like a layer that lives in the same WebGL context as the main canvas. Maybe it's good to have WebGL in the name to make it clear that it only ever can be used for WebGL. I've seen some libraries use the term "surface" to describe something that can be drawn on, although would it be confusing adding anther term like "surface" in addition to the existing but different concepts of "graphics" and "canvas"?
7. I think coming with its own might make the API simpler? That way we wouldn't have to adapt existing camera calls to the framebuffer's size, or adapting the current camera properties to the framebuffer's size when the user binds a framebuffer after changing a camera. Graphics have their own camera systems too so I think it would be consistent with how those work, although it would mean having a fbo.setCamera() method or alternatively knowing that setCamera will set on whatever the current draw target is. Do you think that tradeoff would make sense? A little more onus put on the user but a little less "magic"/surface area for bugs?
8. I think this makes sense!

[aferriss]

HALF_FLOAT makes sense. Do you think it should be a separate option people should pick or something we try to fall back to if FLOAT isn't available?

I think we should probably default to uint8 and allow users to choose another type if they want. If we want to take inspiration from openFramworks we could do something like this

let fbDefault = createFramebuffer(100, 100); // creates a fbo with 8888 int type at the specified size
let fbDefault2 = createFramebuffer(100, 100, RGBA); // creates fbo with the 8888 int type at the specified size
let fbNoAlpha = createFramebuffer(100, 100, RGB);// creates fbo with 888 int type at the specified size
let fbRedOnly = createFramebuffer(100, 100, R); // creates fbo with 8bit int type at the specified size
let fbHalfFloat = createFramebuffer(100, 100, HALF_FLOAT); // creates fbo with 16bit half float type at the specified size
let fbFloat = createFramebuffer(100, 100, FLOAT); // creates fbo with 32 bit float type at the specified size

Alternatively we could allow the function to accept an object which might make parsing settings a little easier. We can default to something sane if the field is left empty on the settings object.

let fboSettings = {
  width: 100,
  height: 100,
  type: FLOAT,
  channels: RGBA
}
let fb = createFramebuffer(fboSettings);

I'm open to a friendlier name, do you have any suggestions?

Nothing especially coming to mind, just wanted to flag it in case we thought of something compelling.

Re:cameras Do you think that tradeoff would make sense?

Makes sense to me, thanks for explaining!

[ShivangMishra]

@davepagurek @aferriss awesome discussion... I've been using p5.js for some of my personal projects, so I'm familiar with how things work in p5. I'd love to contribute to this feature. Any suggestions on where I should start?

[davepagurek]

Thanks for your interest @ShivangMishra! I think the plan for this one will be something along these lines:

• First get Add WebGL2 support #6035 merged to add WebGL2 support
• Then I'll make a PR with initial framebuffer support, to provide some scaffolding for the rest of the features we want
• From there, I'll make some issues for follow-up tasks, including things like reading pixels, ensuring antialiasing works on all platforms with all combinations of settings, writing documentation, and writing examples

Once we get to that third step I'm hoping it'll be easy to pick up tasks and help bring this feature to completion! In the mean time, feel free to keep an eye on this issue to see how things are shaping up, and maybe take a look at p5.Framebuffer, p5Fbo, or openFrameworks ofFbo to get a sense of the sorts of ways we'll want to use this feature.

[ShivangMishra]

@davepagurek alright thanks, I'll go through the related docs and code while you do the first two tasks.

[davepagurek]

Thanks for your patience everyone! Once #6072 is merged, here are a number of follow-up issues that we'd love some help on:

• Verify that p5.Framebuffer settings work in all environments #6083
• Add a way to read back pixels from p5.Framebuffer #6082
• https://github.com/processing/p5.js-website/issues/1340
• https://github.com/processing/p5.js-website/issues/1341