Make the WebGL material system more easy to make libraries for

[davepagurek]

Increasing Access

There have been a number of requests related to the material system in p5, such as adding fog (#5878) or blend modes for ambient light (#6123), I've been working on a library for shader-based warping (https://github.com/davepagurek/p5.warp), and a GSoC project this year will involve working on image-based lighting as an alternative to point/directional/spot lights.

We intentionally don't add every feature into p5 core in order to keep the codebase maintainable, keep the API simple for beginners, and keep the runtime reasonably fast. It would be great to allow community libraries to fill these needs instead! However, the system is currently very difficult to add to externally; the only viable option right now is to package a p5 shader and distribute that, which means keeping your shader up-to-date with internal changes.

A dedicated way to hook into the material system would help people who are interested in contributing via a library test out their ideas, and would give users a larger variety of tools for different needs as new libraries are added.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature request details

The main difference between adding a material to p5 and writing a full shader is that for the former, you generally want to keep most of the existing shader. The best way to do that right now is copy-and-pasting, which goes stale over time and requires expertise of p5's internals to do in the first place. The design goal would be to allow people to replace specific parts of our shaders without needing to do that.

To narrow the scope, I think this only needs to apply to our fill material with lighting, not lines or text for now.

Some potential pieces a library might want to replace are:

• The positioning of vertices onscreen (and updating their normals accordingly) (e.g. the p5.warp library)
• All of the color calculation in the fragment shader (e.g. make some shader texture and just use that)
• Just the color calculation that combines the different lighting terms (e.g. ambientMode: Proposal to prepare an application method other than addition of ambientColor #6123 or add fog #5878)
• The calculation of the lighting terms, which then get combined in the usual way (e.g. to use image-based lighting)

Shader snippets

We can maybe think of our shaders as a collection of code snippets for both the fragment and vertex shader, which have two parts: a header (to specify inputs) and a body (which runs in main()), combined like this:

const shaderSource = `
  precision highp float;

  ${snippets.map(snippet => snippet.head).join('\n')}

  void main() {
    ${snippets.map(snippet => snippet.body).join('\n')}
  }
`;

If we break down our current shaders into snippets like that, then we could provide a minimal API for creating a new shader where one could replace just one part. Maybe something like:

const myMaterial = buildMaterial({
  vertexSnippets: [defaultPositioning],
  fragmentSnippets: [
    defaultDiffuseSpecular,
    myFogSnippet
  ]
})

Some downsides with this are the fact that it treats all snippets just as strings, so there may be naming collisions or type mismatches when making snippets work together. It would at least require relatively minimal code to implement, though.

Shader graph

There's this existing library for combining shader pieces to make one shader: https://www.npmjs.com/package/@gerhobbelt/shadergraph This does much of what the above snippet idea does, but in a much more complete but heavy way, where one can define snippets for small bits of code and build a complicated dependency graph to compile into a shader.

Using this benefits from not being built from scratch, but also adds a new dependency to p5, and means providing a more complicated API to library builders.

Providing access to default shader source

The barest-bones solution maybe just involves exposing the source code for our current material shaders via variables that libraries can reference. That way they could use our existing vertex shader but write their own fragment shader.

This doesn't solve the problem where one wants to use most of our lighting calculations (and therefore be able to integrate with point/directional/spot light calls made in p5) but would still be helpful.

[RandomGamingDev]

I agree that this would be great for general shader use and for #6276

[davepagurek]

I've started making a proof-of-concept system for making a shader graph out of snippets. There's some explanations in the readme here: https://github.com/davepagurek/shader-builder

The things I'm hoping to address with the prototype:

• how one might replace pieces of an existing shader
• how to support different WebGL/GLSL versions + extensions
• whether functions are sufficient abstractions for snippets, and how well we can extract inputs/outputs automatically (I've added some minimal support for destructuring struct types to allow multiple return values from one snippet function)

Some things I still want to think about/tinker with:

• is there a way to minimally package default shaders without adding too many dependencies? (right now it includes a GLSL parser library)
• how coarse or fine should the internal snippets be, and how many should we commit to exposing?
• can snippets come with functions to auto-bind values to them from p5 if not specified otherwise by users?

Let me know if anyone has thoughts so far!

[limzykenneth]

I want to review/rework how addon libraries work to a certain extend as part of the investigations I'm doing. Might be worth thinking about how this fits in there as well.

[davepagurek]

Definitely! What aspects of library building are you thinking about currently?

[limzykenneth]

I'm thinking whether to change how addons are authored for p5.js where currently the general advice is to attach methods directly to the p5 object's prototype (which is all good, jQuery for example does the same) or to also provide a utility function that can help register relevant things such as events or built in hooks (if and when we add them).

A set of minor pros and cons for the prototype approach is that for pro, it is easy to write, align with how internal modules work, and work with existing addons; a con is that addons need to be loaded before p5 initialization and features cannot be dynamically added when the runtime is running.

Also reviewing this as part of the docs review of creating_libraries.md I'm looking at. Some of the things might not be relevant but I'm still thinking.