WebGL2RenderingContext cannot handle large ArrayBuffer sizes.

[terop2]

Please include the following in your bug report:

Version of emscripten/emsdk:
Please include the output emcc -v here
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 2.0.26 (5af6a11)
clang version 13.0.0 (https://github.com/llvm/llvm-project 31e75512174e1bdaa242ee5c7f30fe56e68c3748)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/terop/cvs/emscripten/emsdk/upstream/bin

Failing command line in full:
If this is compile or link-time failure please include the full failing command
along with its entire output.

Full link command and output with -v appended:
Even for runtime issues it helps a lot if you can include the full link command.
Adding -v to the link command will show all of the sub-commands run which
can help us diagnose your issue.
link_command_line.txt

The Problem:
WebGL2 fails to render large meshes. Especially, if -s MAXIMUM_MEMORY=4096MB -s ALLOW_MEMORY_GROWTH=1 is used. Basically when emscripten resizes the heap to go over 2Gb limit, WebGL2 gives this error message:
RangeError: Failed to execute 'bufferData' on 'WebGL2RenderingContext': The ArrayBuffer/ArrayBufferView size exceeds the supported range.,RangeError: Failed to execute 'bufferData' on 'WebGL2RenderingContext': The ArrayBuffer/ArrayBufferView size exceeds the supported range.

In this use case, we're trying to render large 3d models, i.e. sanmiguel 3d model. The code is loading (via fetch) a large 661Mb binary 3d model file, and it fails to render. You can see the issue in https://meshpage.org/431 The error happened immediately after heap resize went over 2Gb limit.

[kripken]

I believe this may be a current browser limitation. cc @kenrussell

[terop2]

The -s MAXIMUM_MEMORY=4096Mb is some new feature in emscripten which is supposed to allow larger heap sizes, but when the rendering context crashes when larger arraybuffer is used, the new memory feature isn't helping as much as we wanted.

[warvstar]

This is fixed in Chrome Canary btw, however the issue persists with glTexImage2D.

I believe the fix was from chromium/chromium@51a8530#diff-e94ddd5e894bc74e9852557fff1f2f85e30ea3507e3b58c6c9c2cd9cdb4a7f15 when BufferSourceTypeNoSizeLimit was added as an attribute type to bufferData (among others)

so the fix for glTexImage2D will likely need to include the same change, for example
void texImage2D(GLenum target, GLint level, GLint internalformat, GLsizei width, GLsizei height, GLint border, GLenum format, GLenum type, [AllowShared] ArrayBufferView srcData, GLuint srcOffset);
to
void texImage2D(GLenum target, GLint level, GLint internalformat, GLsizei width, GLsizei height, GLint border, GLenum format, GLenum type, [AllowShared, BufferSourceTypeNoSizeLimit] ArrayBufferView srcData, GLuint srcOffset);

on the emscripten side, there is a pixels >> heapAccessShiftForWebGLHeap(heap) inside glTexImage2D, that should also probably be >>> when over 2gb?

Edit: Oh and also, these errors only happen when calling the garbage-free entry points for webgl2, the ones noted in library_webgl.js

[kripken]

Thanks @warvstar , yes, the shifts there look like they need updating. I opened #17832 for that now. Please test it if you can, it should fix this issue (together with the chrome canary fix you mentioned).

[terop2]

I can test this once the fixes are available in the emscripten's public repo. With minor configuration change, I can make meshpage.org's sanmiguel test eat so much memory that it will go over 2Gb limit. So once the fixes are somewhat available in the emscripten builds, I'll make the config change and run some 2Gb heap resize tests. (I also figured out some optimizations that allows me to run it without 2Gb heap)

[boardend]

Seems to be related to / a duplicate of #15416

I have the same issue with my WebGL 2 app that throws an error with texImage2D once the memory grows beyond 2GB.

[Narendera-abhiwan]

Hey guys, I just wanted to ask for a solution to a problem that is very similar to what you have already discussed

I am facing this error

Please let me know if you have any solution to it.

[terop2]

Narendra Kumar Jha: Given that filling 2GB of memory takes more than 10 seconds which web page users are not too keen on waiting, best solution to the problem is to use less memory. But sadly the debugging tools in browsers are only giving 2GB block when debugging emscripten memory areas, so the memory debug in emscripten is next to useless.(hint to emscripten, the browser memory area debugging features should be fixed) We've used c++'s operator new/operator delete to debug our memory usage and got significant fixes, i.e. the same 3d model is only stored once in the cpu memory and then it's passed to gpu side when all cpu processing has been done in smaller pieces. But opengl's data flow and c++'s types/sizeof(T) in c++ puts some problems to that area, so every project on the planet cannot do those fixes, and those are only specific to graph-based 3d engines.

I don't currently have a solution (since we're still waiting for newer emscripten build) to the 2GB memory limit problem, and there's a suspect that all webgl apis wont immediately start supporting the 2GB memory areas since it would require larger rewrite for emscripten. But hope is there that this eventually get fixed when people submit bug reports about their 2gb adventures.

[kenrussell]

Andreas Haas from Chrome's V8 team has just landed changes lifting the 2 GB limit for WebGL entry points in Chromium under crbug.com/1344047 . We think this will address these problems, and encourage you to please test with Chrome Canary in the coming days to verify that this is working.

[Narendera-abhiwan]

Narendra Kumar Jha: Given that filling 2GB of memory takes more than 10 seconds which web page users are not too keen on waiting, best solution to the problem is to use less memory. But sadly the debugging tools in browsers are only giving 2GB block when debugging emscripten memory areas, so the memory debug in emscripten is next to useless.(hint to emscripten, the browser memory area debugging features should be fixed) We've used c++'s operator new/operator delete to debug our memory usage and got significant fixes, i.e. the same 3d model is only stored once in the cpu memory and then it's passed to gpu side when all cpu processing has been done in smaller pieces. But opengl's data flow and c++'s types/sizeof(T) in c++ puts some problems to that area, so every project on the planet cannot do those fixes, and those are only specific to graph-based 3d engines.

I don't currently have a solution (since we're still waiting for newer emscripten build) to the 2GB memory limit problem, and there's a suspect that all webgl apis wont immediately start supporting the 2GB memory areas since it would require larger rewrite for emscripten. But hope is there that this eventually get fixed when people submit bug reports about their 2gb adventures.

Thank you finally I got some clue on this, Moreover can you please tell me in a bit of detail how could I optimize this or reduce the size of memory use as I don't have much experience with unity. Should I reduce the texture size or prevent using dynamically allocated variables that I think are allocated inside the heap? or there is another way? I am really really stuck in this.

[terop2]

Narendera: Texture sizes are good optimization, then texture compression is another. Then vertex arrays/mesh sizes should be kept small. That's pretty much it. For unity engine, they should only keep one copy of each asset in the memory, so copying the objects that contain either vertex arrays or textures should be optimized.

[superlucker]

Same here, when can it be fixed? This issue happens only when using webgl2, webgl is totally fine. I also tried Chrome canary and firefox as well, but no luck.

[mehmet-soyturk-qonic]

Maybe related: https://bugzilla.mozilla.org/show_bug.cgi?id=1850507