Stencil support: List of use cases and ideas

[Windfisch]

I think Godot should offer support for stencil buffers to avoid rendering overhead or do fancy effects. However, there are many ways to use them, and it might be difficult to add this feature so it's both easy to use and flexible enough.

The following should become a collection of ideas what the stencil buffer can/will be used for, and what things must be respected in its implementation. Please feel free to add ideas and to discuss :).

---

I'll start here:

• Support for portal-like games: One camera A for the player and another camera B inside a viewport with "the environment on the other side of the portal". That "other" camera B should only render the scene where A can actually see a portal.
• This only requires one bit depth in the stencil, the ability to render objects into the stencil only (setting or clearing bits there), and the ability to do a simple on/off stencil test in the camera.

Possible problem with any implementation: The Godot engine itself may want to use the stencil buffer for its own effects in the future (?). Maybe we can only expose a part of the buffer?

[reduz]

My take on Stencil is that no matter if we eventually add low level support for it, we should support the high level use cases out of the box, then decide whether its worth supporting a low level API.

[reduz]

Use cases that come to mind:

• Removing silhouettes of some objects (clipping them out of geometry using masks).
• Shadow volumes (may not be worth, but some modern games like Mario Odyssey support them for the character)

[antoniotorresm]

The most basic use case is outlining meshes. The current way of outlining meshes in Godot is quite hacky and doesn't get good results (even the example in the docs shows seams between the mesh and the outline).

Another cool use case is to do x-ray vision, like in Overwatch. (Note how back faces are also shown).

Of course, you can use it for "masking" objects. With this you can apply an scene-wide post-process shader, but only to certain objects (those that write to stencil).

These are the main use cases that come to my mind, but there are a lot more. UE4 even has an extension for it called 'Custom depth' which is super useful (you can find some effects there).

[CptPotato]

It's also useful for VR rendering to mask out areas that are not seen on the HMD. For example the OpenVR api can retrieve a stencil mask (if the device/manufacturer provides one).

[Windfisch]

• Removing silhouettes of some objects (clipping them out of geometry using masks).
  you mean like this? https://www.youtube.com/watch?v=9pWFCji7StY

• Shadow volumes (may not be worth, but some modern games like Mario Odyssey support them for the character)
  I think that shadow volumes might be a very special case; I'd rather not support them than to have no, or a too complex, stencil support exposed to the user. (Of course, if there's a nice way to support them, that's great).
  But let's keep them in mind.

Masking objects sounds great; I assume we'll need to set single bits in the stencil for this, at least if we want to support multiple post-proc-effect types.

As for outlining objects, there's a short article here: https://rdmilligan.wordpress.com/2017/08/27/outlining-objects-with-opengl-stencil-testing/. tl;dr: Draw the object into the scene and also into the stencil buffer. Then draw the mesh again, but 1.2x larger, in the desired border color, but only where the stencil hasn't been set (i.e. don't overlap the actual object). This would require a strict sorting of objects, does Godot already support this (for non-transparent objects, and other than "the order in which they appear in the scene tree")?

[reduz]

Outlining and X-Ray using Stencil is overkill IMO and most of these effects mentioned can be done much better adding custom g-buffer support (plannig that for the Vulkan renderer).

This is why I mean, that Stencil makes the most sense to me for masking objects or creating shadow volumes, and not much else.. or effects like Planar Reflections or Portals. Other than that, I think their use is very limited.

[io12]

I am working on a seamless portal Godot asset and stencil buffering would be very helpful for performance!

Not familiar stencil buffers, but it is looking like a way to do many of the effects I'd be puzzling over for a long while. There is a lot of situations where I have a 2D puppet character made of dozens of small sprites, but I want to put a directional gradient over them, or create a selection highlight.

I would need the final combined texture just for those nodes to do this, otherwise a material on all those sprites would only apply to itself, with undesired results.

[clayjohn]

@avencherus It sounds like you need a Viewport, not a stencil. Stencil buffers are used for masking effects. If you want to apply a gradient or an outline to a group of Sprite you should use a Viewport.

[Banduck]

@clayjohn
But the problem with viewports is: when you scale the window, viewports become pixelated. This can't be the best solution.

@clayjohn Thanks, I've heard this to be quite expensive, have not had a chance to profile something like that on dozens of characters. And as Banduck suggests it probably has other quality side effects.

The best thing I've thought of trying is having a script keep track of the sprites, then frame by frame using the VisualServer to draw them onto a new canvas using their textures, ordering, and transform information. Then attach the canvas material to that.

Haven't yet gotten to a point where I will need to experiment with it yet. No idea what the performance would be, but all solutions so far seem like they're doing the work the engine has already done all over again.