Skip to content

Environment Map Filtering GPU pipeline #19076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 59 commits into from
Jul 23, 2025

Conversation

mate-h
Copy link
Contributor

@mate-h mate-h commented May 5, 2025

Objective

This PR implements a robust GPU-based pipeline for dynamically generating environment maps in Bevy. It builds upon PR #19037, allowing these changes to be evaluated independently from the atmosphere implementation.

While existing offline tools can process environment maps, generate mip levels, and calculate specular lighting with importance sampling, they're limited to static file-based workflows. This PR introduces a real-time GPU pipeline that dynamically generates complete environment maps from a single cubemap texture on each frame.

Closes #9380

Solution

Implemented a Single Pass Downsampling (SPD) pipeline that processes textures without pre-existing mip levels or pre-filtered lighting data.

Single Pass Downsampling (SPD) pipeline:

  • accepts any square, power-of-two cubemap up to 8192 × 8192 per face and generates the complete mip chain in one frame;
  • copies the base mip (level 0) in a dedicated compute dispatch (copy_mip0) before the down-sampling pass;
  • performs the down-sampling itself in two compute dispatches to fit within subgroup limits;
  • heavily inspired by Jasmine's prototype code.

Pre-filtering pipeline:

  • generates the specular Radiance Map using bounded-VNDF GGX importance sampling for higher quality highlights and fewer fireflies;
  • computes the diffuse Irradiance Map with cosine-weighted hemisphere sampling;
  • mirrors the forward-/reverse-tonemap workflow used by TAA instead of exposing a separate white-point parameter;
  • is based on the resources below together with the “Bounded VNDF Sampling for Smith-GGX Reflections” paper.

The pre-filtering pipeline is largely based on these articles:

The forward-/reverse-tonemap trick removes almost all fireflies without the need for a separate white-point parameter.

Previous work: #9414

Testing

The reflection_probes.rs example has been updated:

  • The camera starts closer to the spheres so the reflections are easier to see.
  • The GLTF scene is spawned only when the reflection probe mode is active (press Space).
  • The third display mode (toggled with Space) shows the generated cubemap chain.
  • You can change the roughness of the center sphere with the Up/Down keys.

Render Graph

Composed of two nodes and a graph edge:

Downsampling -> Filtering

Pass breakdown:

dowsampling_first_pass -> dowsampling_second_pass ->
radiance_map_pass -> irradiance_map_pass
render-graph

Showcase

image

User facing API:

commands.entity(camera)
    .insert(GeneratedEnvironmentMapLight {
        environment_map: world.load_asset("environment_maps/pisa_specular_rgb9e5_zstd.ktx2"),
        ..default()
    });

Computed Environment Maps

To use fully dynamic environment maps, create a new placeholder image handle with Image::new_fill, extract it to the render world. Then dispatch a compute shader, bind the image as a 2d array storage texture. Anything can be rendered to the custom dynamic environment map.
This is already demonstrated in PR #19037 with the atmosphere.rs example.

We can extend this idea further and run the entire PBR pipeline from the perspective of the light probe, and it is possible to have some form of global illumination or baked lighting information this way, especially if we make use of irradiance volumes for the realtime aspect. This method could very well be extended to bake indirect lighting in the scene.
#13840 should make this possible!

Notes for reviewers

This PR no longer bundles any large test textures.

@alice-i-cecile alice-i-cecile added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen M-Needs-Release-Note Work that should be called out in the blog due to impact labels May 5, 2025
@alice-i-cecile alice-i-cecile added this to the 0.17 milestone May 5, 2025
@alice-i-cecile alice-i-cecile added S-Needs-Review Needs reviewer attention (from anyone!) to move forward D-Complex Quite challenging from either a design or technical perspective. Ask for help! labels May 5, 2025
@JMS55 JMS55 self-requested a review May 6, 2025 00:14
@JMS55
Copy link
Contributor

JMS55 commented May 6, 2025

Some thoughts on the user API:

  • I think I'd like to stick to the old GenerateEnvironmentMapLight name rather than FilteredEnvironmentMapLight.
  • Similarly, we should give users control over when to re-generate the envmaplight. Maybe a boolean that auto-resets on extract like we did for the TAA reset.
  • Does it require using LightProbe? Can you use it for a global envmaplight attached to the camera?

@JMS55
Copy link
Contributor

JMS55 commented May 6, 2025

image

Ran it through NSight.

  1. 0.17 * 6 = 1.02ms of vkCmdCopyBufferToImages 😬
  2. 0.05ms vkCmdCopyImage that I believe is from this PR (no labels 😢)
  3. 0.03ms for SPD in two passes
  4. 0.33ms for radiance map generation. Has good occupancy to start, but drops off towards the end. The workgroups for the smaller mips, I guess? Bottlenecked by texture reads of course.
  5. 0.14ms for irradiance map generation. Very low occupancy the whole time. Very low active threads per warp. I'm not really sure why.

Generally those buffer to image copies are way too slow, need to figure out what those are and remove them. Radiance pass looks good, and irradiance pass isn't that slow but has very suspiciously low active threads per warp.

@JMS55
Copy link
Contributor

JMS55 commented May 6, 2025

Based on the shader profiler, the reason irradiance map is performing poorly is the giant switch statement absolutely kills perf as every thread gets masked out. Should 100% index into an array for that one.

@JMS55
Copy link
Contributor

JMS55 commented May 6, 2025

Oh, I see that you're also doing a separate dispatch per mip for the radiance pass. Does each level depend on the previous?

@janhohenheim
Copy link
Member

This fixes #19125

@mate-h
Copy link
Contributor Author

mate-h commented Jul 21, 2025

After a discussion this morning, we agreed to add the spatio temporal blue noise texture to Bevy PBR plugin. This is from Electronic Arts' fastnoise repo. https://github.com/electronicarts/fastnoise/

I combined the texture using the following command:

wget https://github.com/electronicarts/fastnoise/blob/main/noise.zip
unzip noise.zip && cd noise
ktx create \
      --format R8G8B8_UNORM \
      --assign-tf linear \
      --layers 32 \
      vector3/temporal/exp/vector3_uniform_box3x3_exp0101_product_*.png \
      stbn.ktx2
Screenshot 2025-07-20 at 11 02 46 PM

I did a quick sanity check and decided that the best option going with box3x3. This is the texture most path tracers and ReSTIR implementations ship with today. even for the filtering, I read exactly one texel per pixel, so aliasing that the Gaussian blur fights is not an issue. Box3x3 gives the cleanest high-frequency spectrum and fastest convergence.
Current and future use cases:

  • realtime environment map filtering GGX
  • solari path tracing and ReSTIR GGX
  • raymarching algorithm jittering for volumetrics, atmosphere

Since this step is complete, no remaining actions needed for this PR. ready for a final review and merge.

@alice-i-cecile alice-i-cecile added this pull request to the merge queue Jul 21, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 21, 2025
@alice-i-cecile
Copy link
Member

CI is failing; please take a look!

@mate-h
Copy link
Contributor Author

mate-h commented Jul 22, 2025

Fixed the runtime issue on linux with the vulkan backend. I got a working version of web build locally, switching the bindings around a bit. Since the web support was not in the initial scope of the PR but now including it so the CI can pass on merge. I will create a separate branch and review the changes merging into this branch since everything here has already been reviewed and approved.

@mate-h mate-h mentioned this pull request Jul 22, 2025
@mate-h
Copy link
Contributor Author

mate-h commented Jul 23, 2025

Got feedback from Jasmine that I should try to keep the original setup with the full bind group for the native build, and rely on feature flags - I will attempt to make these changes in the PR to this branch linked above without making a mess.

@mate-h
Copy link
Contributor Author

mate-h commented Jul 23, 2025

@alice-i-cecile this is ready for a merge again. The CI should pass for web and linux.

@alice-i-cecile alice-i-cecile added this pull request to the merge queue Jul 23, 2025
Merged via the queue into bevyengine:main with commit 7b866ee Jul 23, 2025
32 checks passed
@mockersf
Copy link
Member

this PR broke WebGL2: #20276

@mockersf
Copy link
Member

this PR also added an unconditional 1.5MB embedded file to everyone enabling bevy_pbr.

I think it shouldn't have been merged without this being more discussed, and with a way to avoid it.

I'll revert it unless we have a credible plan to fix both issues (webgl2 and size)

@atlv24 atlv24 mentioned this pull request Jul 25, 2025
github-merge-queue bot pushed a commit that referenced this pull request Jul 27, 2025
# Objective

- #19076 (comment)
broke webgl2 and bloated binary size by a megabyte
- Fixes #20276

## Solution

- Don't embed stbn.ktx2, just load it for now
- Gate gen env maps by limit checks
- Extract a plugin to do this cleanly
- Only load stbn.ktx2 when its needed

## Testing

- reflection_probes example
- someone please test webgl2

---------

Co-authored-by: Máté Homolya <mate.homolya@gmail.com>
mate-h added a commit to mate-h/bevy that referenced this pull request Jul 28, 2025
# Objective

- bevyengine#19076 (comment)
broke webgl2 and bloated binary size by a megabyte
- Fixes bevyengine#20276

## Solution

- Don't embed stbn.ktx2, just load it for now
- Gate gen env maps by limit checks
- Extract a plugin to do this cleanly
- Only load stbn.ktx2 when its needed

## Testing

- reflection_probes example
- someone please test webgl2

---------

Co-authored-by: Máté Homolya <mate.homolya@gmail.com>
tychedelia pushed a commit to tychedelia/bevy that referenced this pull request Jul 31, 2025
# Objective

- bevyengine#19076 (comment)
broke webgl2 and bloated binary size by a megabyte
- Fixes bevyengine#20276

## Solution

- Don't embed stbn.ktx2, just load it for now
- Gate gen env maps by limit checks
- Extract a plugin to do this cleanly
- Only load stbn.ktx2 when its needed

## Testing

- reflection_probes example
- someone please test webgl2

---------

Co-authored-by: Máté Homolya <mate.homolya@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible D-Complex Quite challenging from either a design or technical perspective. Ask for help! M-Deliberate-Rendering-Change An intentional change to how tests and examples are rendered M-Needs-Release-Note Work that should be called out in the blog due to impact S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Automatic Skybox -> EnvironmentMapLight generation
8 participants