Fix lightmaps break when deferred rendering is enabled

[Soulghost]

Objective

• Fixes Lightmaps break when deferred rendering is enabled #13552

Solution

• Thanks for the guidance from @DGriffin91, the current solution is to transmit the light_map through the emissive channel to avoid increasing the bandwidth of deferred shading.
• Store lightmap sample result into G-Buffer and pass them into the Deferred Lighting Pipeline, therefore we can get the correct indirect lighting via the apply_pbr_lighting function.
• The original G-Buffer lacks storage for lightmap data, therefore a new buffer is added. We can only use Rgba16Uint here due to the 32-byte limit on the render targets.

Testing

• Need to test all the examples that contains a prepass, with both the forward and deferred rendering mode.
• I have tested the ones below.
  • lightmaps (adjust the code based on the issue and check the rendering result)
  • transmission (it contains a prepass)
  • ssr (it also uses the G-Bufffer)
  • meshlet (forward and deferred)
  • pbr

Showcase

By updating the lightmaps example to use deferred rendering, this pull request enables correct rendering result of the Cornell Box.

diff --git a/examples/3d/lightmaps.rs b/examples/3d/lightmaps.rs
index 564a3162b..11a748fba 100644
--- a/examples/3d/lightmaps.rs
+++ b/examples/3d/lightmaps.rs
@@ -1,12 +1,14 @@
 //! Rendering a scene with baked lightmaps.
 
-use bevy::pbr::Lightmap;
+use bevy::core_pipeline::prepass::DeferredPrepass;
+use bevy::pbr::{DefaultOpaqueRendererMethod, Lightmap};
 use bevy::prelude::*;
 
 fn main() {
     App::new()
         .add_plugins(DefaultPlugins)
         .insert_resource(AmbientLight::NONE)
+        .insert_resource(DefaultOpaqueRendererMethod::deferred())
         .add_systems(Startup, setup)
         .add_systems(Update, add_lightmaps_to_meshes)
         .run();
@@ -19,10 +21,12 @@ fn setup(mut commands: Commands, asset_server: Res<AssetServer>) {
         ..default()
     });
 
-    commands.spawn(Camera3dBundle {
-        transform: Transform::from_xyz(-278.0, 273.0, 800.0),
-        ..default()
-    });
+    commands
+        .spawn(Camera3dBundle {
+            transform: Transform::from_xyz(-278.0, 273.0, 800.0),
+            ..default()
+        })
+        .insert(DeferredPrepass);
 }
 
 fn add_lightmaps_to_meshes(

Emissive Issue

The emissive light object appears incorrectly rendered because the alpha channel of emission is set to 1 in deferred rendering and 0 in forward rendering, leading to different emissive light result. Could this be a bug?

// pbr_deferred_functions.wgsl - pbr_input_from_deferred_gbuffer
let emissive = rgb9e5::rgb9e5_to_vec3_(gbuffer.g);
if ((pbr.material.flags & STANDARD_MATERIAL_FLAGS_UNLIT_BIT) != 0u) {
    pbr.material.base_color = vec4(emissive, 1.0);
    pbr.material.emissive = vec4(vec3(0.0), 1.0);
} else {
    pbr.material.base_color = vec4(pow(base_rough.rgb, vec3(2.2)), 1.0);
    pbr.material.emissive = vec4(emissive, 1.0);
}

// pbr_functions.wgsl - apply_pbr_lighting
emissive_light = emissive_light * mix(1.0, view_bindings::view.exposure, emissive.a);

[Olle-Lukowski]

Btw, maybe we should open another issue for the emissive issue?

[JMS55]

Related: #13871

[DGriffin91]

The output of the deferred pass is already too large. We can't afford to include an additional attachment. I think the lightmap data should be stored in the emissive channel as it is essentially emissive.

[Soulghost]

The output of the deferred pass is already too large. We can't afford to include an additional attachment. I think the lightmap data should be stored in the emissive channel as it is essentially emissive.

Good idea, the emissive object seems not need a lightmap, let me do some research and change the code.

[Soulghost]

If we use the deferred.y to store emissive or lightmap, we need to add an extra bit to distinguish between them. Therefore we can no longer use rgb9e5::vec3_to_rgb9e5_ to pack the emissive value, we need to pack it with the pack_unorm4x8_, there maybe loss of precision, is it affordable? Perhaps we should enable it only when a lightmap is included.

Note that the pack_unorm4x8_ can only store color values between 0.0 to 1.0. Therefore, if the emissive component's value exceeds one, as in the EDR rendering, this presents a problem.

Previous

let deferred = vec4(
    deferred_types::pack_unorm4x8_(vec4(base_color_srgb, in.material.perceptual_roughness)),
    rgb9e5::vec3_to_rgb9e5_(emissive),
    props,
    deferred_types::pack_24bit_normal_and_flags(octahedral_normal, flags),
);

New

let deferred = vec4(
    deferred_types::pack_unorm4x8_(vec4(base_color_srgb, in.material.perceptual_roughness)),
    deferred_types::pack_unorm4x8_(vec4(emissive_or_lightmap, has_lightmap)),
    props,
    deferred_types::pack_24bit_normal_and_flags(octahedral_normal, flags),
);

[DGriffin91]

Both the emissive and the light map need the extra precision and range from rgb9e5. My recommendation is actually to sum the emissive and light map into the emissive channel as light maps are essentially emissive.

You'd want to pre-multiply it by the diffuse color as seen here:

bevy/crates/bevy_pbr/src/render/pbr_functions.wgsl

Line 551 in 0c7df88

indirect_light += in.lightmap_light * diffuse_color;

[Soulghost]

Hi @DGriffin91 , I have used the emissive channel to transmit the lightmap data. Could you please review the code? Thank you.

[Soulghost]

Hi @JMS55 , I have synchronized my code with upstream and made necessary adaptations for the changes in the emissive's alpha channel. Now the lightmap works perfectly without the emissive issues we discussed earlier.

[Soulghost]

Hi @DGriffin91 , could you please help review these changes? Thank you.

[IceSentry]

Code looks fine, but can you answer me and @JMS55 about the LIGHTMAP shader_def that is still in the code instead of quietly resolving? I don't get why it's there.

[Soulghost]

Hi @IceSentry and @JMS55 , we need to keep the LIGHTMAP shader definition because the fragment shader defined in pbr.wgsl is invoked by the Prepass Pipeline. The lightmap data is derived through the pbr_input_from_standard_material function. It is essential to use the LIGHTMAP keyword to ensure the lightmap_light is correctly fetched.

// Invoked from Prepass Pipeline
fn pbr_input_from_standard_material(
    in: VertexOutput,
    is_front: bool,
) -> pbr_types::PbrInput {
    // ...
    #ifdef LIGHTMAP
        pbr_input.lightmap_light = lightmap(
            in.uv_b,
            pbr_bindings::material.lightmap_exposure,
            in.instance_index);
    #endif
    // ...
}