Sky rendering with transparent viewport

[BastiaanOlij]

Godot version

4.2.dev d5217b6

System information

N/a

Issue description

Currently when you switch a viewport to transparent and a sky is setup (which it often is by default) we still render the sky but we now render it with ALPHA set to 0.

The first and obvious issue with this is that still rendering the sky is wasteful, while there is no indication to the user that this is still happening.

The second less obvious issue with this is that we may want to render the sky.

The use case here is for MR, where we need to keep parts of the viewport transparent so we can see the real world, but where we may still want to see the sky, for instance so the user can still see their desk, or their hands, or other real world objects.
This would be accomplished by rendering opaque geometry that write a zero alpha to the render buffer to erase whatever was rendered there before, and prevent rendering of sky in the subsequent sky pass (this may require further enhancements though we added support for this in Godot 3 for ARKit, so this would be a regression if no longer possible).

The only scenario where a transparent sky could be a valid use case is if we don't want the background, but we do want sky contribution for ambient lighting (i.e. radiance map) but there are better ways to make this work.

Steps to reproduce

• Create a new project
• Create a main scene (3d)
• Add a world environment with a sky
• Add a Camera3D
• Add a script with the following code:

func _ready():
  get_viewport().transparent_bg = true

See the sky disappear, now check with renderdoc or something similar, and see the sky actually still being rendered but with Alpha = 0.

Minimal reproduction project

Not yet, will make something soon.

[thygrrr]

The sky has two functions, one is the rasterized skybox (skysphere), and one is for the irradiance maps in lighting and reflections.

This means if you keep your sky active in the environment, you can RIGHTFULLY expect the irradiance to be calculated as per your sky shaders.

If you don't select the half or quarter resolution render modes, then yes, your irradiance cubemap is going to be rendered at full resolution.

If the final blit is being rasterized and blended for alpha = 0, then sure, that should not execute.

I believe processing "invisible" skies is also important for scenes where the background is another camera / environment, and you need the irradiance for the foreground as well (e.g. planet shine in space scenes)

[BastiaanOlij]

@thygrrr the problem is that rendering the sky to the viewport, and producing the irradiance map are two separate processes part of the same hole.

If for instance you are running a full physical sky, rendering the sky to the viewport with alpha 0, is a LOT of overhead for no output. If you need the irradiance map, what we're missing is a way to skip the rendering of the background without skipping updating the irradiance map.

My problem however is that I need to sky to render, which is currently not possible. I'm masking out other areas that will remain transparent but the sky will be visible outside of those masked areas. So the easiest solution for now is to add a flag whether the sky should be rendered with an alpha of 0, and then except the unnecessary overhead we already except. But it feels to me the real requirement here is to have an option to just update the irradiance maps and skip the rest of the sky logic.

@clayjohn mentioned that the alpha 0 is also needed for SSS but it wasn't clear to me whether this was again because we need the irradiance map to be updated, or whether we actually need the color data in the viewport. It would be good to have more background info recorded on this.

[BastiaanOlij]

cc @dsnopek , this one will become important for you soon. This change will also be important to allow us to do things like what Apple is doing on the Vision Pro. Being able to make the background semi transparent or transparent in certain areas, so that the camera image will shine through.

This will require being able to set the alpha on the sky. A related issue here that we need to solve is that right now, setting ALPHA output in a shader results that shader being executed in the transparent pass and the ALPHA being used to blend with the current value. For XR we're going to need the capability to write to alpha during the opaque pass so we can erase stuff. This will allow us to do stuff like render geometry for the users desk, or render geometry where the users hands are, so that those areas stay transparent and show the camera image.

[BastiaanOlij]

Note, we chose to solve the issue through #91642 and skip sky rendering on transparent backgrounds. It still allows for the sky to create reflections.

If you do need a sky background while transparent bg is on, you'll need to render the sky on a sphere yourself.

[Zireael07]

@BastiaanOlij This should be documented

[BastiaanOlij]

@Zireael07 hmm, wonder what a good place for that would be..