Environment Map Filtering GPU pipeline

[mate-h]

Objective

This PR implements a robust GPU-based pipeline for dynamically generating environment maps in Bevy. It builds upon PR #19037, allowing these changes to be evaluated independently from the atmosphere implementation.

While existing offline tools can process environment maps, generate mip levels, and calculate specular lighting with importance sampling, they're limited to static file-based workflows. This PR introduces a real-time GPU pipeline that dynamically generates complete environment maps from a single cubemap texture on each frame.

Closes #9380

Solution

Implemented a Single Pass Downsampling (SPD) pipeline that processes textures without pre-existing mip levels or pre-filtered lighting data.

Single Pass downsampling (SPD) pipeline:

• takes the 512x512 Cubemap as an input, and creates 9 MIP levels.
• each level is progressively down-sampled.
• It is actually broken into two passes due to architectural limitations.
• largely based on Jasmine's code from github

Pre-filtering pipeline: composed of multiple Radiance Map (specular mips) generation passes, followed by the irradiance map pass (diffuse).
The pre-filtering pipeline is largely based on these articles:

• https://placeholderart.wordpress.com/2015/07/28/implementation-notes-runtime-environment-map-filtering-for-image-based-lighting/
• https://bruop.github.io/ibl/

Interesting note: the fireflies are almost completely gone by using the forward tonemap and reverse tonemap trick, while setting the whitepoint to 1.0. If the white point is set to higher, the fireflies come back. This is only apparent for envrironment maps with "hot" spots (i.e. values close to the maximum).

Previous work: #9414

Testing

The reflection_probes.rs example has been enhanced with:

• A fourth display option (toggled via spacebar)
• Adjustable roughness for the center sphere (using Up/Down keys)

First test case and use-case, we load a KTX texture with the RGBA 16-bit floating point format. I obtained this texture from polyhaven.com and used the ./convert.sh goegap_road_2k.exr command to convert it to a cubemap.

I had to install 4 different command line tools to get this to work (not ideal): OpenEXR CLI, ImageMagick, OpenImageIO, and finally the KTX command line tool. And I lose half the precision in this process going from 32-bit input to 16-bit output image.

convert.sh

#!/bin/bash
# create cubemap from equirectangular
file_name=$1
# remove the extension
file_name=$(basename "$file_name" .exr)
exrenvmap -c -li -w 512 -m -z none "$file_name.exr" "cubemap_%.exr"
echo "Created cubemap from equirectangular"

# fix the exr files with imagemagick
files=cubemap_*.exr
for file in $files; do
    magick "$file" "$file"
    oiiotool "$file" --fixnan box3 -o "$file"
    echo "Processed $file"
done

rm "$file_name.ktx2" > /dev/null 2>&1

ktx create --format R16G16B16A16_SFLOAT \
    --assign-tf linear \
    --cubemap \
    --zstd 3 \
    "cubemap_+X.exr" "cubemap_-X.exr" "cubemap_+Y.exr" "cubemap_-Y.exr" "cubemap_-Z.exr" "cubemap_+Z.exr" \
    "$file_name.ktx2"
echo "Created $file_name.ktx2"
ktx info "$file_name.ktx2" | grep "vkFormat"

---

Showcase

User facing API:

commands.spawn((
    LightProbe,
    FilteredEnvironmentMapLight {
        environment_map: world.load_asset("environment_maps/goegap_road_2k.ktx2"),
        ..default()
    },
    Transform::from_scale(Vec3::splat(2.0)),
));

Computed Environment Maps

To use fully dynamic environment maps, create a new placeholder image handle with Image::new_fill, extract it to the render world. Then dispatch a compute shader, bind the image as a 2d array storage texture. Anything can be rendered to the custom dynamic environment map.
This is already demonstrated in PR #19037 with the atmosphere.rs example.

We can extend this idea further and run the entire PBR pipeline from the perspective of the light probe, and it is possible to have some form of global illumination or baked lighting information this way, especially if we make use of irradiance volumes for the realtime aspect. This method could very well be extended to bake indirect lighting in the scene.
#13840 should make this possible!

Notes for reviewers

This PR includes a 7.2 MB KTX file for testing, this can of course be removed if it adds too much weight in the repo. We could include this "magic script" I came up with somewhere in the codebase as well. Robswain@ was suggesting that we implement the cubemap generation in a compute pipeline as well, so we could use equirectangular env maps directly instead (future work).

[JMS55]

What is this needed for?

[JMS55]

Might be a little big, maybe shrink the resolution a bit?

[JMS55]

I'd like for bevy to include some spatiotemporal blue noise from https://github.com/electronicarts/fastnoise.

We shouldn't use the temporal aspect for this feature, but I'd like to use a STBN texture now so we don't have multiple copies of blue noise.

[JMS55]

I don't like calling this a prefilter envmap. The "EnvironmentMap" option is also prefiltered, just offline :P

[JMS55]

Do we want to hardcode to 512x512? 256x256 might be cheaper and not worse quality? Or maybe we should let the user configure.

[JMS55]

What happens if the original cubemap is not 512x512?

[JMS55]

What's the white point calculation here? It's different from what I used in TAA, where was the math derived from? And is it meant to be user-configurable?

[JMS55]

Array indexing or even indexing into a buffer might make more sense here? Idk.

[JMS55]

Assuming this is what I think it is, I suggest using https://github.com/JMS55/bevy/blob/solari6/crates/bevy_solari/src/scene/sampling.wgsl#L89-L97.

If you want to be cool you can move it into some shared file I can later use in bevy_solari :)

[JMS55]

Why does this matter?

[JMS55]

FYI this uses the lambert BRDF, which is not what Bevy uses for StandardMaterial.

Generally I don't think it matters, we probably already assume lambert elsewhere, but you could technically importance sample the other BRDF if you did some more math.

[JMS55]

Some thoughts on the user API:

• I think I'd like to stick to the old GenerateEnvironmentMapLight name rather than FilteredEnvironmentMapLight.
• Similarly, we should give users control over when to re-generate the envmaplight. Maybe a boolean that auto-resets on extract like we did for the TAA reset.
• Does it require using LightProbe? Can you use it for a global envmaplight attached to the camera?

[JMS55]

I forget which is which, but we need to be careful of using roughness vs perceptual_roughness in this code.

[JMS55]

Ran it through NSight.

1. 0.17 * 6 = 1.02ms of vkCmdCopyBufferToImages 😬
2. 0.05ms vkCmdCopyImage that I believe is from this PR (no labels 😢)
3. 0.03ms for SPD in two passes
4. 0.33ms for radiance map generation. Has good occupancy to start, but drops off towards the end. The workgroups for the smaller mips, I guess? Bottlenecked by texture reads of course.
5. 0.14ms for irradiance map generation. Very low occupancy the whole time. Very low active threads per warp. I'm not really sure why.

Generally those buffer to image copies are way too slow, need to figure out what those are and remove them. Radiance pass looks good, and irradiance pass isn't that slow but has very suspiciously low active threads per warp.

[JMS55]

Based on the shader profiler, the reason irradiance map is performing poorly is the giant switch statement absolutely kills perf as every thread gets masked out. Should 100% index into an array for that one.

[JMS55]

Oh, I see that you're also doing a separate dispatch per mip for the radiance pass. Does each level depend on the previous?