wasm64: We can't yet use wasm memories > 4Gb

[sbc100]

The first issue is that HEAPU8 (type array view) cannot yet be larger then 4Gb: https://bugs.chromium.org/p/v8/issues/detail?id=4153

The second issue is that WebAssembly.memory.grow doesn't seem to allow for growing over 4Gb. See other.test_failing_growth_wasm64

[TobiasNoell]

Can this be fixed / bypassed by another emscripten implementation or is this a fundamental issue within chromium?

[sbc100]

There ard two issues here the max typed array sise issue is fairly fundamental... We did have some idea about how we could stitch together multiple views to present a single fake HEAPU8 but it likely a fair bit of work and it would have a runtime cost.

[sbc100]

Regardless you wouldn't be able to ship your app today since wasm64 still depends on a chrome flag. Is there some reason you are interested in getting this working today?

[TobiasNoell]

Our use case would be with Node on the backend, so this additional flag would not be a problem for us.

[sbc100]

I see, that is interesting. Unrelated to this issue, but I've always been curious to find out if folks were actually using emscripten output under node in production. Could you give any more details of your use case?

[TobiasNoell]

Yes, we are 😉 we for example are using wasm to load EXR image files using the TinyEXR library. The image files can get quite huge from times (> 10000x10000 pixel, float), exceeding the 4GB memory limit.

[sbc100]

Any why are you choosing to use a WebAssembly of TinyEXR? Why not just use a native build?

[TobiasNoell]

We also support loading EXRs in the frontend, so we use the same WASM implementation also in the backend (we compile our WASM library for backend and for frontend)

[TobiasNoell]

But maybe allow me for a follow up question: How would one be able to load a larger than 4GB into a javascript array in Node by "just using a native build"?

[sbc100]

I think that answer to you last question is yes, but I'm not totally sure. My understanding is that the current limitation is on the size of types array view. If you use array buffers directly, or data views, I don't think the same limit applies.

Regarding using the same code on the frontend and backend, I think that seems a like a compelling an useful use case. If you would be interested in testing out >4Gb access as we develop our solutions that would be very useful I think.

[TobiasNoell]

Thanks for the clarification.

Sure, I would definitely be interested in testing > 4GB memory allocation once support was added to emscripten!

[jspanchu]

Regarding testing, we're exploring 64-bit support in the VTK library because we've users who want to render massive 3D models in the browser. I'd be glad to offer up VTK as a test case for >4GB support. Currently, we've hit an issue with SDL2 which I've filed in libsdl-org/SDL#8241

[fraguada]

Checking in on this as we also have a back-end / front end use case. We currently compile for front-end but have users that have large 3d models that they want to process with node and are reaching the memory limit. We'd then compile with MEMORY64 and run with the experimental flag. Is this working for node use cases?

ref mcneel/rhino3dm#512

[TobiasNoell]

Is this working for node use cases?

From the test I have been doing so far: No. But maybe something has changed since I last tested (June).
There seemed also to be some very fundamental restrictions inside Node / V8. E g. it is currently not possible to use > 32 bit indexes for arrays.

[sbc100]

This should work now since we added polyfill/proxy for larger array views: #19737

Also, the upstream issue in v8 has (I believe) now been fixes so recent canary builds of node should not even need the polyfill, and we should be able to remove it completely one day.

[sbc100]

This issue has now been fixes and the polyfill has been removed.