Large Meshes render incorrectly (depth buffer issue)

[npip99]

Godot version: v3.2.alpha0.official

OS/device including version: Ubuntu 18.04 LTS

Issue description: If a mesh is exceptionally large, on the order of 100k-1M in each direction, and it is centered on the camera, then is it culled weirdly and will pop in and out of existence as you look around.

Steps to reproduce: Create a new PlaneMesh, with scale 1,000,000, centered at 0,0. Have the camera centered at 0,0, with far distance set to 90,000. When you run the scene, it'll be glitchy because the plane will be popping in and out of view.

(This was happening to me with the terrain, which understandably made the game unplayable). A fix is to have four meshes, each having a corner at 0,0, so that they're making up each quadrant of the map. So for some reason, the issue only occurs when you're at the center of the mesh, not when you're at the corner.

Minimal reproduction project:
godotbug.zip

[akien-mga]

I can reproduce it on Linux with AMD Radeon RX Vega M, both with GLES2 and GLES3, on 3.1.1-stable and current master (3ca1296).

For me it seems to be quite deterministic (in this demo), there are two intervals of Y rotation angles for which the clipping happens.

[lawnjelly]

If you set your camera near clipping plane from 0.01 to 1, the problem doesn't seem to occur. I don't think it's the octree culling code that is misbehaving, more something due to lack of precision in camera space.

Edit, for more info have a look at this:
https://developer.nvidia.com/content/depth-precision-visualized

Note that with standard z, precision is non-linear and very sensitive to the near plane setting.

There is also a reversed z mapping trick which can help with precision issues, I don't know if Godot has this available.

[npip99]

It looks like lawnjelly has a solution if it's possible to implement reversed z. One thing I thought of is simply changing the minimum near distance. It's already bounded at 0.01 but if 0.01 causes bugs and 0.05 doesn't then the real minimum should be 0.05 until reversed z is implemented, unless reversed z is trivial to implement and you might as well fix that first (idk anything about this to make that call)

As a note, the default camera distance is currently 100 but if 0.05 is bugless at infinitely far depths I see no reason not to make the default far plane infinite and default nearplane the minimum one that works; this a common technique that keeps the game dev outside of the implementation details as much as possible, since it's not really "more expensive" to have the near and far planes any farther than the farthest possible distance. Unless you count frustum culling but tbh unexpected frustum culling in the far plane just means the game dev will have to update the far draw distance anyway since parts of the scene would be literally disappearing without any confirmation that the game dev wanted that.

[Raphael2048]

In OpenGL, reversed-z need OpenGL4.5 later or ARB_clip_control support to use glClipControl, and GLES didn't support this. So it could be very complicated to achieve this.

[lawnjelly]

In OpenGL, reversed-z need OpenGL4.5 later or ARB_clip_control support to use glClipControl, and GLES didn't support this. So it could be very complicated to achieve this.

Yep I noticed this too in the linked doc. So would be confusing to support multi-platform if some supported this and others didn't.

@npip99 I think it is difficult to set defaults for near and far plane that will work for everyone, you usually have to set them according to your game. There are also other tricks for rendering things far into the distance, e.g.
https://godotforums.org/discussion/21313/how-to-simulate-very-far-away-objects-camea-far-clipping-distance

[npip99]

@lawnjelly I mean it's hard for me to discuss because I really don't know much about this, but I was referencing "An infinite far plane makes only a miniscule difference in error rates. Upchurch and Desbrun predicted a 25% reduction in absolute numerical error, but it doesn't seem to translate into a reduced rate of comparison errors.", which seems to imply that an infinite far plane is just as good. So it seems making the default infinite makes sense. But like again I don't really get the details because I'm not familiar with this topic. Reading that article helped a lot though, thanks, I fixed a lot of bugs I was having with my water and planes getting glitchy and I had no idea it was because of depth buffer issues. Maybe an explanation of this phenomenon should be in the gd docs? That should be enough to close this issue because if we can't avoid the bug by the nature of how z buffers work, then we'd have to explain it to new users.

[lawnjelly]

To be a little more specific about this particular issue, I can say what I think might be going on, but this is all conjecture, I haven't investigated in any detail, it is just a theory based on the behaviour, and I'm going out on a limb here. There may be an alternative much better explanation. 😄

Generally the effect of near and far clipping planes is to determine how much z fighting goes on when triangles are rendered 'close together' in the z coordinate (in projected camera space, rather than world space), i.e. the resolution of the z buffer is the issue, and that is what the linked article is mostly concerned with.

In this particular case, rather than the resolution, I suspect that the issue may be due to the coordinates somewhere in the pipeline being way outside the expected range, and the nature of floating point calculations. I.e. the near and far plane don't just determine the final resolution, they determine the actual z values of vertices in clip and NDC space that are outside the camera range.

As I understand it the vertices in geometry in godot (and 3d apps in general) tend to go through one or more transformation matrices that get them into clip space just before they get rendered:
https://learnopengl.com/Getting-started/Coordinate-Systems

I expect that this pipeline is made with the assumption that the triangles coming through the system with reasonable values. Once the vertices are being transformed way outside this range, the contract is broken and weird things might be happening. You would hope the system would deal with such numbers gracefully, but ultimately the pipeline will be built for speed rather than dealing with special cases.

In practice I doubt this occurs much except in such test cases, because even with a large terrain, you would typically tessellate it. You could add a special path in rendering code (or GPUs) to deal with such things, but I expect it would be ridiculous to do so because it would probably slow down all other rendering for a case which would hardly ever happen.

[Skaruts]

I think I'm having the same problem (or seemingly closely related), and I think @lawnjelly may be half-right. I say half-right because, at least in my case, this only happens in GLES2 (in GLES3 everything is just fine). However, as I bumped up z-near in small steps it did seem to make the problem happen further and further away from the camera, and at some point I couldn't tell it was there anymore, so... I guess that kinda "fixes" it.

This is what it looks like on my end (these meshes aren't that big, the plane is 200x200, and the cube is 20x20x20:

[Calinou]

Maybe GLES2 uses a 16-bit depth buffer, which makes precision issues much more visible?

Also, note that on Intel IGPs, depth buffers will be 16-bit unless you explicitly request 24-bit precision (unlike AMD and NVIDIA GPUs, where the default is to use a 24-bit depth buffer).

[Skaruts]

@Calinou Hmm, well, the thing is, it seems to render fine under some circumstances. I was just now thinking of this: adding a SpatialMaterial to the cube and turning on one of these flags makes it render fine against the plane: Transparent, Unshaded or No-Depth-Test. Weirdly enough, with no materials, when I select the cube it suddenly renders fine.

(Just by the way, unfortunately I didn't think to give a different color to the cube before recording the gif, so it's not noticeable there that the cube actually pops in and out of existence at times, much like the OP mentioned happening to his meshes.)