Skip to content

[Native] Honor depthCullingState.depthTest on the native engine#18558

Merged
bkaradzic-microsoft merged 2 commits into
masterfrom
native-honor-depthcullingstate-depthtest
Jun 16, 2026
Merged

[Native] Honor depthCullingState.depthTest on the native engine#18558
bkaradzic-microsoft merged 2 commits into
masterfrom
native-honor-depthcullingstate-depthtest

Conversation

@bkaradzic-microsoft

@bkaradzic-microsoft bkaradzic-microsoft commented Jun 10, 2026

Copy link
Copy Markdown
Member

[Native] Honor depthCullingState.depthTest on the native engine

Problem

EffectRenderer (used by EffectWrapper-based fullscreen passes) disables depth
testing by mutating the shared depth-culling state directly:

// Materials/effectRenderer.pure.ts -> applyEffectWrapper()
this.engine.depthCullingState.depthTest = depthTest; // false for a fullscreen pass

On the WebGL/WebGPU engines this is flushed to the GPU by applyStates(), which is
called from drawElementsType/drawArraysType and reads depthCullingState.

The native engine does not go through that applyStates() path — its draw methods
encode a draw command directly, and depth state is only ever changed through the
explicit setDepthBuffer() / setDepthFunction() commands. As a result, a depth-test
toggle done through engine.depthCullingState.depthTest never reaches the native
command stream.

The visible effect: a fullscreen EffectWrapper quad keeps the depth-test state of the
previously rendered geometry (e.g. LEQUAL). Drawn at clip-space z = 0.5 against a
render target whose depth buffer is not at the far value, every fragment fails the depth
test and is discarded, so the pass produces an all-black target even though the pixel
shader output is correct.

ThinDepthPeelingRenderer toggles depthCullingState.depthTest directly as well and is
affected by the same gap.

Fix

Make the native engine honor depthCullingState.depthTest the same way the other
engines do:

  • setDepthBuffer() keeps _depthCullingState.depthTest in sync (matching the base
    AbstractEngine.setDepthBuffer, which is literally this._depthCullingState.depthTest = enable).
  • Before each native draw, _flushDepthTestState() reconciles the encoded depth-test
    enable with depthCullingState.depthTest and emits the command only when it changed.

This keeps the existing explicit setDepthBuffer() / setDepthFunction() paths working
unchanged (they already set the state object / are the source of truth) while also
honoring direct depthCullingState.depthTest mutations.

No public API change.

Testing

Validated on Babylon Native (Win32 / D3D11) with the Playground validation suite:

  • The previously-broken EffectRenderer onion-skin/blur fullscreen-pass scene renders
    correctly instead of all black.
  • Full standard validation sweep: no regressions (every previously-passing test still
    passes, including depth- and transparency-sensitive scenes).

Notes

EffectRenderer also toggles stencilState.stencilTest directly. That has the same
latent gap on native but defaults to false and has no reproducing scene today, so it is
intentionally left out of this change to keep it minimal; it can be addressed with the
same pattern if a repro appears.


Partial fix for BabylonJS/BabylonNative#1106 (Handle Engine States in Babylon Native).

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a Babylon Native rendering gap where toggling engine.depthCullingState.depthTest (notably by EffectRenderer / EffectWrapper fullscreen passes) was not reflected in the native command stream, causing depth-test state to “leak” from previous draws and potentially discard fullscreen fragments.

Changes:

  • Add native-side tracking for whether depth testing is currently encoded/enabled, and reconcile it before each draw.
  • Update setDepthBuffer() to keep the shared _depthCullingState.depthTest in sync and encode the corresponding native command.
  • Flush depth-test enable/disable state at the start of drawElementsType / drawArraysType so direct depthCullingState.depthTest mutations are honored.

Comment thread packages/dev/core/src/Engines/thinNativeEngine.pure.ts Outdated
Comment thread packages/dev/core/src/Engines/thinNativeEngine.pure.ts
@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Snapshot stored with reference name:
refs/pull/18558/merge

Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18558/merge/index.html

To test a playground add it to the URL, for example:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18558/merge/index.html#WGZLGJ#4600

Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves):

https://playground.babylonjs.com/?snapshot=refs/pull/18558/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/18558/merge
https://gui.babylonjs.com/?snapshot=refs/pull/18558/merge
https://nme.babylonjs.com/?snapshot=refs/pull/18558/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/18558/merge#BCU1XR#0

If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools.

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

🟢 Memory Leak Test Results

13 passed, 0 leaked out of 13 scenarios

🟢 All memory leak tests passed — no leaks detected.

Passed Scenarios (13)
Scenario Package
Core Feature Stack @babylonjs/core
Core Rendering Materials Shadows Stack @babylonjs/core
Core Textures Render Targets PostProcess Stack @babylonjs/core
GUI Fullscreen UI Controls @babylonjs/gui
GUI Mesh ADT Controls @babylonjs/gui
Loaders Boombox Import @babylonjs/loaders
Loaders OBJ Direct Load @babylonjs/loaders
Loaders STL Direct Load @babylonjs/loaders
Materials Library Stack @babylonjs/materials
Serializers glTF Export @babylonjs/serializers
Serializers GLB Export @babylonjs/serializers
PostProcesses Digital Rain Stack @babylonjs/post-processes
Procedural Textures Stack @babylonjs/procedural-textures

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

⚡ Performance Test Results

🟢 All performance tests passed — no regressions detected.

@bghgary bghgary left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Reviewed by Copilot on behalf of @bghgary]

Targeted fix for one slice of #1106 (native engine not honoring state set through the property objects) — worth linking there. The open setDepthFunction thread is a real gap in the new _flushDepthTestState tracking; should be closed or consciously deferred like the stencil case.

bkaradzic-microsoft pushed a commit to bkaradzic-microsoft/BabylonNative that referenced this pull request Jun 12, 2026
This EffectRenderer fullscreen-pass test rendered all-black on Native because the
native engine did not honor depth-test toggles made through
engine.depthCullingState.depthTest (used by EffectRenderer). Fixed upstream in the
native engine (BabylonJS/Babylon.js#18558); re-enable the test here.

Depends on a babylonjs dependency bump that includes the fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
EffectRenderer (and ThinDepthPeelingRenderer) disable depth testing by setting
engine.depthCullingState.depthTest directly. On WebGL/WebGPU this is flushed by
applyStates() at draw time, but the native engine only honored depth state set
through the explicit setDepthBuffer() command and never flushed depthCullingState
before a draw, so the toggle was silently dropped. A fullscreen EffectWrapper pass
then kept the previous draw's depth test and every fragment was discarded, producing
an all-black target.

Reconcile depthCullingState.depthTest before each native draw to match the
cross-engine contract: setDepthBuffer() keeps the state object authoritative and
_flushDepthTestState() emits the command when the direct toggle differs from what was
last encoded. No public API change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bkaradzic-microsoft bkaradzic-microsoft force-pushed the native-honor-depthcullingstate-depthtest branch from 59d94a6 to 889cfb1 Compare June 12, 2026 22:43
Route setDepthFunction() through _encodeDepthTest() so the tracked
_depthTestEnabled state always matches the encoded COMMAND_SETDEPTHTEST.
Previously setDepthFunction encoded the compare function directly, which
(since DEPTH_TEST_ALWAYS doubles as "disabled") could silently re-enable
depth testing while it was logically disabled, leaving _flushDepthTestState
unable to reconcile the desync before the next draw.

Also initialize _stencilTest to false (it was a no-op statement), so
getStencilBuffer() returns a proper boolean before setStencilBuffer runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bkaradzic-microsoft

Copy link
Copy Markdown
Member Author

Thanks @bghgary — addressed both points in 7d90b87:

  • setDepthFunction desync: closed (not deferred). It now routes through _encodeDepthTest(this._depthCullingState.depthTest) so the tracked _depthTestEnabled always matches the encoded command stream, and setDepthFunction no longer silently re-enables depth testing when it is logically disabled (DEPTH_TEST_ALWAYS doubles as the disabled state).
  • stencil init no-op: _stencilTest is now initialized to false, so getStencilBuffer() returns a proper boolean before setStencilBuffer() runs.

This PR is a targeted slice of BabylonJS/BabylonNative#1106 (Handle Engine States in Babylon Native) — linked there as well.

@bjsplat

bjsplat commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

@bjsplat

bjsplat commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Snapshot stored with reference name:
refs/pull/18558/merge

Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18558/merge/index.html

To test a playground add it to the URL, for example:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18558/merge/index.html#WGZLGJ#4600

Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves):

https://playground.babylonjs.com/?snapshot=refs/pull/18558/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/18558/merge
https://gui.babylonjs.com/?snapshot=refs/pull/18558/merge
https://nme.babylonjs.com/?snapshot=refs/pull/18558/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/18558/merge#BCU1XR#0

If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools.

@bkaradzic-microsoft bkaradzic-microsoft enabled auto-merge (squash) June 12, 2026 23:46
@bjsplat

bjsplat commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

🟢 Memory Leak Test Results

13 passed, 0 leaked out of 13 scenarios

🟢 All memory leak tests passed — no leaks detected.

Passed Scenarios (13)
Scenario Package
Core Feature Stack @babylonjs/core
Core Rendering Materials Shadows Stack @babylonjs/core
Core Textures Render Targets PostProcess Stack @babylonjs/core
GUI Fullscreen UI Controls @babylonjs/gui
GUI Mesh ADT Controls @babylonjs/gui
Loaders Boombox Import @babylonjs/loaders
Loaders OBJ Direct Load @babylonjs/loaders
Loaders STL Direct Load @babylonjs/loaders
Materials Library Stack @babylonjs/materials
Serializers glTF Export @babylonjs/serializers
Serializers GLB Export @babylonjs/serializers
PostProcesses Digital Rain Stack @babylonjs/post-processes
Procedural Textures Stack @babylonjs/procedural-textures

@bjsplat

bjsplat commented Jun 13, 2026

Copy link
Copy Markdown
Collaborator

@bjsplat

bjsplat commented Jun 15, 2026

Copy link
Copy Markdown
Collaborator

@sebavan

sebavan commented Jun 16, 2026

Copy link
Copy Markdown
Member

cc @CedricGuillemet or @bghgary for review please :-)

@bghgary bghgary left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Reviewed by Copilot on behalf of @bghgary]

LGTM

@bkaradzic-microsoft bkaradzic-microsoft merged commit a736e85 into master Jun 16, 2026
21 checks passed
@bkaradzic-microsoft bkaradzic-microsoft deleted the native-honor-depthcullingstate-depthtest branch June 16, 2026 23:35
bkaradzic-microsoft pushed a commit to bkaradzic-microsoft/BabylonNative that referenced this pull request Jun 18, 2026
This EffectRenderer fullscreen-pass test rendered all-black on Native because the
native engine did not honor depth-test toggles made through
engine.depthCullingState.depthTest (used by EffectRenderer). Fixed upstream in the
native engine (BabylonJS/Babylon.js#18558); re-enable the test here.

Depends on a babylonjs dependency bump that includes the fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bkaradzic-microsoft pushed a commit to bkaradzic-microsoft/BabylonNative that referenced this pull request Jun 18, 2026
This EffectRenderer fullscreen-pass test rendered all-black on Native because the
native engine did not honor depth-test toggles made through
engine.depthCullingState.depthTest (used by EffectRenderer). Fixed upstream in the
native engine (BabylonJS/Babylon.js#18558); re-enable the test here.

Depends on a babylonjs dependency bump that includes the fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bkaradzic-microsoft pushed a commit to bkaradzic-microsoft/BabylonNative that referenced this pull request Jun 18, 2026
This EffectRenderer fullscreen-pass test rendered all-black on Native because the
native engine did not honor depth-test toggles made through
engine.depthCullingState.depthTest (used by EffectRenderer). Fixed upstream in the
native engine (BabylonJS/Babylon.js#18558); re-enable the test here.

Depends on a babylonjs dependency bump that includes the fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants