`RenderingDevice` - `texture` functions are no longer allowed in background threads due to thread guards.

[RPicster]

Tested versions

• Reproducible in 4.4 (all dev versions)
• Works reliable in 4.3

System information

Godot v4.4.dev5 - Windows 10.0.19045 - Single-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 4080 (NVIDIA; 32.0.15.6094) - AMD Ryzen 7 3700X 8-Core Processor (16 threads)

Issue description

A small intro: I tried talking to @clayjohn and @BastiaanOlij about this behaviour, but so far I couldn't find out anything that enables me to move forward with it.

The setup in the MRP is what I essentially use in Engraving in different parts of the game to create an Image from a viewport texture in a asynchronous way, without stalling the game like a viewport.get_texture().get_image() would. For a lot of things it's not critical that the image is instantly available but it's more important to have a smooth game experience.

With the release of 4.4 this approach does not work at all anymore and keeps generating error messages, it works reliable in 4.3 stable.

My game is heavily depening on having a smooth way of creating an image from the players drawings to further process this image in real-time, so this change keeps me from updating the version and doing further testing on 4.4 (or even release in 4.4+).

If it is not possible to continue with this approach it would be greatly appreciated to find an alternative way of achieving async image generation from a viewport - which is also useful in a lot of other scenarios (vfx using viewports as textures/buffers).

Steps to reproduce

Checking out the MRP would be the easiest :)

Minimal reproduction project (MRP)

rd_copy_texture.zip

EDIT

After the discussions in this issue, the main challenge appeared to be getting texture / image data from the GPU to the CPU without a stall. It's not a regression per se - how I used to do what I needed to seemed to be a very flawed approach.
The best solution would be a new function, like suggested here by Dario or here by myself.

[BastiaanOlij]

cc @DarioSamo , you might have some insight to this? I haven't had time to fully dive into what is going on here but I imagine we plugged some holes here especially when rendering runs on a separate thread.

[DarioSamo]

cc @DarioSamo , you might have some insight to this? I haven't had time to fully dive into what is going on here but I imagine we plugged some holes here especially when rendering runs on a separate thread.

Yes, the main change that has been made, for thread-safety and correctness reasons, is that a lot of RenderingDevice calls are no longer allowed. The main reason is that RenderingDevice used to do a ton of concurrent work by just doing a Mutex basically everywhere and hoping for the best. This also meant that even if you had what looked like it was asynchronous generation working, it was in fact not asynchronous at all and merely slotting in the recording of the current frame wherever it could and getting done in the current frame, making your frame actually take longer than it should.

True asynchronous GPU work takes advantage of the fact that the hardware offers multiple queues to execute the work and has a much lesser impact on the work produced on the current frame. This is why 4.4 introduces the use of transfer queues that actually run in parallel to the game, which gave a huge improvement to loading speeds in V-Sync'd scenarios.

With the release of 4.4 this approach does not work at all anymore and keeps generating error messages, it works reliable in 4.3 stable.

The problem here is this gives the false appearance that it was working reliably, while in fact it was doing it in a way that introduces a ton of possible validation errors that it is practically impossible for us to assess. GPU Validation is a bad area to risk having problems because the problems are not reliably reproduced and might show up differently on different hardware.

A major change that 4.4 brings is the possibility to actually use different command queues that the driver offers, but RenderingDevice has no API to expose this at the moment. The closest we have is creating a local rendering device, but sadly that does not allow you to share resources with the main one as it does create an entirely new different driver in the background.

I realize this might be perceived as a regression in the eyes of users, but rolling back to the old design would keep hiding away the problem instead of addressing it properly. It was an explicit request by @reduz to introduce thread guards, so I think he should weigh in suggesting the API we should use to provide some functionality like this again to do it properly.

As for unblocking your approach in 4.4 in some way even if it's not the best solution, my suggestion would be to use the mechanism GDScript provides to defer calls to the main render thread. As I said, it was never actually processing the images in an asynchronous way anyway, as all it did was throw the processing that you requested into the main render queue for the current frame (which introduced several race conditions), so you can very much operate with a similar logic that just does deferred calls to the render thread and checks on the data on the next frame. This is what I mean by the fact that it wasn't truly asynchronous anyway, and it was pretty evident because assets loaded in background threads were never actually loaded and had to wait for V-Sync all the time as they'd just get thrown in whatever current frame was active and had to wait it out. The same problem was probably happening with the effect you mention.

[RPicster]

Thanks for looking into it so quickly 👏
For me, the main difference was, that using the method in the MRP resulting in a much smoother experience vs using viewport.get_texture().get_image(). So, async or not - the result for me was the same, trading "getting the image immediately" for "having to stall something".

Could you give me a quick example on how I would implement "defer calls to the main render thread" or a working approach using mechanisms you describe? I can also reach out in the godot chat if that is preferred, but maybe it would be better to have it here in case someone else comes across this problem?

[DarioSamo]

Could you give me a quick example on how I would implement "defer calls to the main render thread" or a working approach using mechanisms you describe? I can also reach out in the godot chat if that is preferred, but maybe it would be better to have it here in case someone else comes across this problem?

I think one big issue is that deferred calls might not actually help much if the use case is 'texture_get_data' because that function in particular has a bit of a design problem, where it has an extremely heavy amount of CPU work from copying the data back to RAM while the GPU itself has to do very little work. So if that function is now locked to being called in the render thread, it'll cause a big stall regardless of us moving to the more correct approach. One approach we discussed with @clayjohn is that functions such as get_data() would benefit from having asynchronous versions that you can call and query at a later point, allowing the data copy to happen independently from the rendering process once the GPU guarantees the texture or buffer is ready for it.

But to give you an example of something that was even worse, by using this function on a background thread, calling any function like get_data() basically requires flushing all of the queued frames of the game, pretty much demolishing any performance whatsoever the render thread was doing in being able to queue frames and keep the GPU busy! These are the kind of problems that not being explicit about what should be kept on the render thread causes, and it's completely hidden away from the user in the older approach.

Perhaps you can explain your problem space a bit better so we can avoid having to do this on the CPU and keep it entirely on the GPU instead. Is there a particular reason you need to download the image to the CPU and manipulate it there? Could it be perhaps achieved through compute shaders? And at what frequency do you need to fetch the viewport image. is it per frame or in an interval of time?

[RPicster]

Perhaps you can explain your problem space a bit better so we can avoid having to do this on the CPU and keep it entirely on the GPU instead. Is there a particular reason you need to download the image to the CPU and manipulate it there? Could it be perhaps achieved through compute shaders? And at what frequency do you need to fetch the viewport image. is it per frame or in an interval of time?

Here is one example:

• The player can draw the map inside a SubViewport
• An image is created of this viewports texture
• The image is converted to a bitmap.
• grow_mask and opaque_to_polygons is then used to create a texture of filled spaces.
• This texture is used in a shader to create the "filling" for the drawing.
  This happens while the player can draw and is saved / loaded when changing the sectors of the map (each sector having it's own map).

Godot_v4.4-dev5_win64_tLvo061xXo.mp4

In another example, visibility splats are generated when the player walks around which uncover the map.

• When the player moves /rotates distance x, a new "splat" is generated, this is a "field of view" capture of their current view.
• This "splat" is then converted to a "static" texture (viewport texture -> image -> texture) and added to another viewport as a sprite.
• The splats will slowly fade out as new splats are added.

[DarioSamo]

@RPicster I can see why you'd go the route of needing CPU manipulation if it's a gameplay mechanic.

Is the entire reason you went with the asynchronous approach because the get_data() routines were causing stalls? In that case it seems like the discussion should be less centered around being able to do rendering commands in a separate thread but rather, being able to request the data asynchronously. As you're clearly able to hide the latency for gameplay reasons, it sounds like you'd be better served by a scheme where you can do this sort of work in the main thread.

request_id = texture_get_data_async_request(); <- this gives you some sort of ID which you can use to query if the data is ready at a later point
... then during process on each later frame ...
texture_get_data_async_query(request_id, image); <- this allows you to query whether the texture's been copied to a data block that you can now copy off and process in a background thread

The main benefit of this approach would be that you no longer need to stall the GPU and wait for frames to be flushed compared to 4.3, which would give you higher performance.

[RPicster]

Yes, you absolutely hit the nail on the head. In the end this was just a solution I found for the problem that is stalling.
something like:

func update_texture():
   viewport.get_texture().get_data_async(image_callback)

func image_callback(image:Image):
   ...