Numeric instability of `world_from_clip` in orthographic projection

[Lichtso]

Bevy version and features

Version: v0.16.1
Only default features

Relevant system information

Rust 1.86.0
macOS 15.6.1
Apple silicon M4 Max

What you did

Using view.world_from_clip * vertex_output.clip_space_position to get a ray origin for some ray tracing in the fragment shader. The last row of view.world_from_clip however is highly unstable in orthographic projections and requires to be patched roughly like so: view.world_from_clip[3].xyz = view.world_position;

What went wrong

I assume the multiplication of inverted matrices in the uniforms leads to this instability. More specifically it appears that view_from_clip is at fault, not world_from_view. Which is to be expected because the near / (near - far) can produce a value far away from all others on the log scale, thus leading to floating point precision loss.

See the screenshots below for what they look like.

Additional information

[Lichtso]

I investigated the calculation of world_from_clip inside the engine and it turns out the matrix is fine. I was just under the wrong impression that the W component would hold the world translation of the camera.

However, because of how the orthograhpic projection (misnomer, because is mathematically not a projection!) works you also need the Z component to cancel out the streching of the frustum in Z direction.

⎡  2 / w       0         0  0 ⎤
⎢      0   2 / h         0  0 ⎥
⎢      0       0     1 / d  0 ⎥
⎣ cw / w  ch / h  near / d  1 ⎦

which does not happen in perspective projection because there is no Z streching, it is flattened:

⎡ f / aspect  0     0   0 ⎤
⎢          0  f     0   0 ⎥
⎢          0  0     0  -1 ⎥
⎣          0  0  near   0 ⎦

For posternity, anybody trying to do raytracing, the correct way is:

// Vertex shader
#ifdef VIEW_PROJECTION_ORTHOGRAPHIC
    vertex_output.clip_position.z = view.clip_from_view[3][2];
#endif

// Fragment shader
#ifdef VIEW_PROJECTION_ORTHOGRAPHIC
    let ray_origin = (view.world_from_clip * vertex_output.clip_position).xyz;
    let ray_direction = -view.world_from_view[2].xyz;
    let near_clipping = view.clip_from_view[3][2] * view.clip_from_view[2][2];
#else // VIEW_PROJECTION_PERSPECTIVE
    let ray_origin = view.world_from_view[3].xyz;
    let ray_direction = (view.world_from_clip * vertex_output.clip_position).xyz;
    let near_clipping = view.clip_from_view[3][2];
#endif