[browser][MT] WebGL with threads - requestAnimationFrame

[pavelsavara]

How we could make webGL (and other native scenarios) possible with MT ?

I believe that the lack of access to the "true" UI thread would completely break WebGL interop.
WebGL doesn't have a dedicated present/swapBuffers call and instead just presents render results once the user code exits the current event callback. If JS API calls are getting enqueued from a worker thread, that would mean that webgl content will be "presented" on every single call, thus making any kind of rendering impossible.

Another problem would be WebGL usage with SkiaSharp: Skia uses WebGL internally and expects those calls to happen on the "true" UI thread rather than on a web worker.

From #85592 (comment)

[pavelsavara]

Few ideas for start

• The JSImport calls to JS could be synchronous, but not re-entrant.
  • Do you need synchronous callbacks, when/why/how ?
• Note that UI thread is and will stay native/emscripten thread.
  • some of sys calls and VFS POSIX calls are synchronously proxied to the UI thread.
  • Perhaps we could make webGL native calls also proxied ?
  • Search proxiedFunctionTable in emscripten codebase.

[pavelsavara]

cc @kekekeks @maxkatz6

[kekekeks]

In general WebGL calls should happen inside of the callback from window.requestAnimationFrame on the browser UI thread. That particular callback timing is required to match the screen update rate.
All WebGL calls related to a single frame should happen inside of a single browser event loop iteration (i. e. inside of said
window.requestAnimationFrame callback).

So the typical scenario would be:

1. JS code from window.requestAnimationFrame sets up rendering
2. JS code synchronously calls .NET code
3. .NET code synchronously calls WebGL functions either via JSImport or emscripten P/Invokes or Skia code that uses emscripten APIs internally. All of those should happen inside of the same window.requestAnimationFrame callback
4. frame rendering is finished
5. .NET code exits and returns control back to JS
6. JS returns from window.requestAnimationFrame callback
7. Browser assumes that we are done rendering and queues render results for presentation

So WebGL rendering requires synchronous JS->.NET->JS calls directly on the main browser thread.

Note that it's not possible to prepare everything as a "call list", since some WebGL calls can read data back from GPU in synchronous manner and we can't interrupt the call sequence by exiting the event callback since the browser would assume that we are done rendering the frame.

[kekekeks]

Note that we don't really need lock()/.Wait on the browser thread (there are currently some short-lived locks by those can be replaced with lock-free code), just a way to call into .NET code and call JS back.

[kekekeks]

Actually, I haven't considered locks inside of SkiaSharp. It utilizes locks extensively for its object handle tracking. Those are rather short lived (and won't even be contended in most apps), so spinlocks should be fine.

[pavelsavara]

WebGL calls should happen inside of the callback from window.requestAnimationFrame on the browser UI thread.
3. .NET code synchronously calls WebGL functions

This is nasty nested synchronous callback.
The the UI thread is spin-waiting for the semaphore from the first call to resolve/return, when you want to deliver the message about the nested call.

The syscalls in the proxiedFunctionTable are exceptions from this rule and will get executed inside of the UI spin-lock.
It's in the middle of unrelated business logic, which is on stack, it could arrive from any other thread.
It's processed kind of "out of order" WRT the current synchronous call.
This is how emscripten VFS works, not pretty, but it "works".

Note that we don't really need lock()/.Wait on the browser thread.

You can try it with

        dotnet.withConfig({
            jsThreadBlockingMode: "ThrowWhenBlockingWait",
        });

or with "DangerousAllowBlockingWait"

which will allow you to make (un-nested) synchronous JSExport calls.

My lessons learned from working on this for last 12 months is that you never know.

• you could be lucky and win the race for the lock
• C# spin-wait for while and then win the race for the lock
• actually engage in real wait
  • we will throw PNSE when ThrowWhenBlockingWait, but it's far from 100% covered.
  • this is to let you learn that you are blocking the managed thread and the UI thread as well
  • chance you see this in test is low
    • one of many of CI runs of unit tests on runtime repo, there are many tests

when UI thread is blocked, event loop is blocked, UI doesn't render, postMessage doesn't work, debugger sucks, new WebWorkers are impossible to spawn.
when deputy thread is blocked it can't receive emscripten messages from other threads.

If there is wait/promise chain this could be deadlock.
How do you know that you are not blocking waiting for HTTP stream in 3rd party code you don't own.
That HTTP promise would never resolve.

All that said, this is not to say "I give up" 😉

Does it have to be managed code, whatever is talking to webGL from requestAnimationFrame callback ?
Could all what's needed be pre-computed on background thread and applied in JS or C ?

Blazor people have renderBatch implemented in just JavaScript. The current problem with that is that they are reading the "diff" directly from managed memory. They also have server side diff message that is applied to DOM. They will have to adopt "diff message" rather than "C# memory scan" to make Blazor MT compatible. More about it here

[pavelsavara]

Also would this help ?
https://github.com/kripken/webgl-worker

Or this
https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/transferControlToOffscreen

We have draft of JSWebWorker API, which allows your C# to execute managed code on new WebWorker and JS interop with state of that WebWorker. Again this is not public API yet and also has the problem with blocking .Wait vs event loop and resolving promises.

[kekekeks]

The the UI thread is spin-waiting for the semaphore from the first call to resolve/return, when you want to deliver the message about the nested call.

Why does the runtime need to marshal the call to a web-worker in the first place? Can't it just execute the code directly on the UI thread? The requirement for WebGL rendering is to specifically run everything rendering-related on the main thread inside of the requestAnimationFrame callback and to take minimum possible time, since the code has to run 240 times per second with 240Hz monitors and be synchronized to VSync.

when UI thread is blocked, event loop is blocked, UI doesn't render, postMessage doesn't work, debugger sucks, new WebWorkers are impossible to spawn.

We only need the UI thread to run our rendering code. It doesn't do any IO nor wait on any long-running semaphores (i. e. spin-waits are fine).

The need to run all of the rendering code directly inside of requestAnimationFrame is a requirement from browsers, not from our architecture.

Could all what's needed be pre-computed on background thread and applied in JS or C ?

Unfortunately, not really, some of WebGL calls are not fire-and-forget but require actual handling of returned values or are supposed to block on reading back from GPU (i. e. if we want to copy some computed data from GPU to CPU mid-frame). Also, we aren't doing most of the calls by ourselves, the usual chain is SkiaSharp > Skia (native) -> emscripten opengl layer -> JS, so even if we don't need readback, we can't even replay the commands on the UI thread without executing managed code there

https://github.com/kripken/webgl-worker

It doesn't allow a huge part of OpenGL calls to work. I. e. it doesn't support reading back from GPU memory.

https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/transferControlToOffscreen

IIRC OffscreenCanvas doesn't support WebGL contexts on some browsers (last time we've checked it didn't work with Safari), only 2d drawing.

If it does work now, the requirements would be mostly the same: we'll need to call into .NET from JS running in a web worker and have callbacks executed in the same worker and without costly cross-worker marshalling, since there are potentially thousands of calls per-frame.

Again this is not public API yet and also has the problem with blocking .Wait vs event loop and resolving promises.

The worker's event loop shouldn't be occupied with anything but requestAnimationFrame and gl context loss/restore callbacks, so it should be fine to have blocking waits there as long as those don't consume CPU time.