-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent background redraw for transparent widget (embedding wgpu inside druid) #1462
Comments
I've just seen #1249 and changed |
The background is drawn at |
Thanks @jneem; no not really, I see a change (it flickers to black instead of dark gray) but there's still some bad interaction between |
Sadly I think this is a composition issue. The way I patched up things, wgpu reuses the same HWND created by Druid:
Anyway I think the two composition pipelines are targeting the same HWND and rendering in parallel, fighting to present to the window. One workaround I am thinking of trying when I have time is to create a child window (HWND) inside the Druid one, and use that one for wgpu. I think that could possibly work around the issue. Otherwise unfortunately I don't see how to do for now, except waiting for piet to have a wgpu backend that is exposed and can be accessed for manual rendering. |
I'm very interested in wgpu, but think that having embedded 3d widgets is a pretty hard problem. Either you use wgpu for everything, which I think is fine (but involves building that), or you start having to expose composition up through the widget hierarchy, so for example a wgpu inside a scroll view can get translated and clipped appropriately as it scrolls. Doing that in a cross-platform way is hard. I'm not sure there's a simple answer to this, or what might be the best path forward. See #891 for more discussion. |
Yes I've read #891 actually before starting my experiment. This is why I was trying with I've looked at how Qt does it nowadays and it is interesting to see they went the child window approach back in 2013 (I couldn't find anything more recent on the subject; I think this is still the recommended approach), which is my next plan for when I have time. This delegates the clipping to the OS compositor by leveraging the parent-child relationship of the native windows (HWND on Windows). Then this native child window (with no border etc.) can be embedded via a custom Druid widget which controls its position and size. In fact the Qt documentation has a list of restrictions in this case, and they explicitly mention the scrolling one (highlights mine):
I know this approach is also used sometimes when e.g. the inner window is rendered by a separate process (for example, a game engine) to avoid crashes to the renderer affecting the stability of the surrounding application. This is doable on Windows for sure, and I believe on other OSes too. So I think this may be a more feasible approach short term, even if it means it has limitations compared to other widgets. I'll see if I can try it when I have some spare time. Would you be open for Druid to take this kind of change? I appreciate this somehow goes against the philosophy of "druid is safe high-level abstraction, druid-shell contains the low-level platform-dependent implementation", since it would leak the native platform handle to end users for them to embed their content (unless the change is restricted to embedding wgpu only, in which case the widget can act as an abstraction barrier). |
Thanks for looking into Qt, I think that's good background. Overall I am open to landing this in Druid, as I think it's good to get experience; it's clearly a use case people care about. I think if we're up front about the limitations and don't let it constrain our development too much, it can be ok. We're already partly down the path of doing subwindows for things like menus. I also think it's a fair amount of work, especially as it has to be rethought on different platforms, and even within Windows, as subwindow is compatible with Windows 7, but "swapchain for composition" might be a richer and more performant experience on 10. Similarly for X11 and Wayland on Linux. But if you're willing to take it on, I certainly won't stop you :) |
I am willing to try it on Windows 10 for now, see if that works at all :) |
@dhardy thank you for the shameless plug of good info! I had a very quick look at Iced, and didn't know KAS. I am going to continue for now with Druid because I really like its design, somewhat (I personally find) in-between event-based UI (Qt style / React) and gaming style "immediate-mode UI" (no state inside widgets). So I'm really curious to see if I can overcome that 3D limitation and try a simple 2D+3D app with that Druid approach of embedding the data inside the widgets. As for progress, I struggled quite some time trying to drill a hole into that druid-shell abstraction, but I think I've finally found a way. No doubt I broke stuffs in the process, but at least I got that: Note how the wgpu native child window renders over its parent correctly, thanks to (I think) This is a prototype; don't mind the wrong aspect ratio, the native child window position/size are hard-coded. The way things roughly work:
|
Update and food for thought:
|
So I implemented the I need to clean-up things a bit and I'll try to at least push a branch on a fork, or even maybe make a draft PR to discuss the code directly. There's only Windows support, but once the bits are in place for child windows I believe this should be easy to replicate for other platforms. EDIT: The gif is animated; it doesn't seem to work in preview, but clicking on it will show the animation. |
I still have some issue to set the origin. I had temporarily overlooked it but now would like to fix it, because this breaks as soon as the wgpu view widget is not a direct child of the root one. The native child window origin needs to be the position of that window relative to its parent window, which in general is not its parent widget but another ancestor. However since Any thought or guidance on this @jneem or @raphlinus ? If it makes more sense with pseudo-code, now I have: parent.layout() {
child.layout();
child.set_origin(origin) {
self.origin = origin;
if is_native {
set_native_origin(child_hwnd, self.origin); //< This is wrong if parent is not a window
}
}
} And I think we'd need instead: parent.layout() {
child.layout();
child.set_origin(origin) {
self.origin = origin;
}
}
parent.post_layout(native_origin) {
native_origin += self.origin;
child.post_layout(native_origin) {
native_origin += self.origin;
if is_native {
set_native_origin(child_hwnd, native_origin); //< Now this is properly accumulated from ancestor window
}
}
`` |
Working version of WgpuView widget with proper native child window and auto-resize, but with invalid native origin. This allows using the widget only if a direct child of the top-level window. bug: linebender#1462
Latest version pushed to fork branch The remaining issue about the native origin is there : djeedai@9d4db19#diff-18ee77c070e42a5dfbf2a1f99a1f1afdeff1a92a5aeb1ebfed22d112154edef6R294 |
What if we were to have the window-relative position available in |
That's a good idea, provided any widget that moves always receives a paint event for that move. I'll see when I can find some time if I can try it that way. |
Something like that maybe? 😜 The gray and coral things are (hacked) 10-px wide paddings. The light gray stuff is the 6-px bar of some split widget. See full example The (very dirty; not ready for PR) commit adding native origin to Note the delay in redraw and the visible positioning artifacts when moving/sizing. This is because accessing the native origin (= window-relative position) in |
This change adds partial support for embedding a wgpu rendering pipeline inside a widget hierarchy via a new `WgpuView` widget backed by a native platform window child of the main top-level window. The following set of changes allow embedding the wgpu pipeline: - Add the `LifeCycleCtx::request_native_window()` method to associate a platform window to a widget during the `WidgetAdded` event. - Add a new event `Event::NativeWindowConnected` raised when the native platform window has been created, to allow a widget to initialize platform resources requiring that child window handle. The event receives a new `NativeWindowHandle` object which allows accessing that child platform window. - Add a new field `PaintCtx::native_origin` which describes the position of the widget relative to its inner-most native platform window, which is almost always the top-level window (`Window<T>`) or can be a child native window when using the previous mechanism. - Add a new method `Widget::post_render()` invoked after the entire widget hierarchy completed a `Widget::paint()` traversal and the rendering back-end submitted the associated rendering commands to the GPU. This allows `WgpuView` or any other similar future widget to submit further draw commands to the GPU (via `wgpu` in the current case) without the risk of a race condition with the normal druid back-end. This mechanism is a workaround for the lack of explicit synchronization or unified rendering pipeline. - Implement the `HasRawWindowHandle` trait for `WindowHandle` to allow native interop to any crate supporting that trait, without explicit dependency to said crate. The new `WgpuView` widget uses those new features to provide a framework to render arbitrary 3D content inside that widget via a new `WgpuRenderer` trait. This widget is only enabled with the new `wgpu_view` feature, because it requires some additional dependencies that users might not be interested in always pulling into their project if not using that widget. A new `wgpu_view` example demonstrates how to use that new widget by embedding a `WgpuView` widget inside multiple `Split` widgets in a configuration similar to common desktop 3D applications (toolbars on the side and top, and 3D view in the center). The sample reuse the cube sample from the `wgpu-rs` project. Known limitations: - The change is limited to support for the Windows platform, although it lays some architecture that should make it easy to add support for other platforms. - This change only add support for single-depth native child windows, that is platform windows which are a direct child of a top-level window. A platform child window cannot be parented to another platform child window. - Interactive animation of the `WgpuView` content (that is, painting outside the `post_render()` callback, for example at a given framerate) is not addressed nor even explored by this change, and may require future work. It is assumed here that the 3D content is static and rendering is entirely druid-event-based (mouse click, window resize, etc.). Bug: linebender#1462
This change adds partial support for embedding a wgpu rendering pipeline inside a widget hierarchy via a new `WgpuView` widget backed by a native platform window child of the main top-level window. The following set of changes allow embedding the wgpu pipeline: - Add the `LifeCycleCtx::request_native_window()` method to associate a platform window to a widget during the `WidgetAdded` event. - Add a new event `Event::NativeWindowConnected` raised when the native platform window has been created, to allow a widget to initialize platform resources requiring that child window handle. The event receives a new `NativeWindowHandle` object which allows accessing that child platform window. - Add a new field `PaintCtx::native_origin` which describes the position of the widget relative to its inner-most native platform window, which is almost always the top-level window (`Window<T>`) or can be a child native window when using the previous mechanism. - Add a new method `Widget::post_render()` invoked after the entire widget hierarchy completed a `Widget::paint()` traversal and the rendering back-end submitted the associated rendering commands to the GPU. This allows `WgpuView` or any other similar future widget to submit further draw commands to the GPU (via `wgpu` in the current case) without the risk of a race condition with the normal druid back-end. This mechanism is a workaround for the lack of explicit synchronization or unified rendering pipeline. - Implement the `HasRawWindowHandle` trait for `WindowHandle` to allow native interop to any crate supporting that trait, without explicit dependency to said crate. The new `WgpuView` widget uses those new features to provide a framework to render arbitrary 3D content inside that widget via a new `WgpuRenderer` trait. This widget is only enabled with the new `wgpu_view` feature, because it requires some additional dependencies that users might not be interested in always pulling into their project if not using that widget. A new `wgpu_view` example demonstrates how to use that new widget by embedding a `WgpuView` widget inside multiple `Split` widgets in a configuration similar to common desktop 3D applications (toolbars on the side and top, and 3D view in the center). The sample reuse the cube sample from the `wgpu-rs` project. Known limitations: - The change is limited to support for the Windows platform, although it lays some architecture that should make it easy to add support for other platforms. - This change only add support for single-depth native child windows, that is platform windows which are a direct child of a top-level window. A platform child window cannot be parented to another platform child window. - Interactive animation of the `WgpuView` content (that is, painting outside the `post_render()` callback, for example at a given framerate) is not addressed nor even explored by this change, and may require future work. It is assumed here that the 3D content is static and rendering is entirely druid-event-based (mouse click, window resize, etc.). Bug: linebender#1462
I opened a PR. It's probably not going to be mergeable as is, but should be a good base for further discussion on the code itself. We can then break down individual changes if needed for easier review and I can make separate PRs for those. There's also the (newly discovered 😥) issue of sub-window seemingly implementing the same thing as I was working on with child native windows, and which was pushed already, so would require more work to be adapted to (if it does what I think it does; I didn't look). |
This change adds partial support for embedding a wgpu rendering pipeline inside a widget hierarchy via a new `WgpuView` widget backed by a native platform window child of the main top-level window. The following set of changes allow embedding the wgpu pipeline: - Add the `LifeCycleCtx::request_native_window()` method to associate a platform window to a widget during the `WidgetAdded` event. - Add a new event `Event::NativeWindowConnected` raised when the native platform window has been created, to allow a widget to initialize platform resources requiring that child window handle. The event receives a new `NativeWindowHandle` object which allows accessing that child platform window. - Add a new field `PaintCtx::native_origin` which describes the position of the widget relative to its inner-most native platform window, which is almost always the top-level window (`Window<T>`) or can be a child native window when using the previous mechanism. - Add a new method `Widget::post_render()` invoked after the entire widget hierarchy completed a `Widget::paint()` traversal and the rendering back-end submitted the associated rendering commands to the GPU. This allows `WgpuView` or any other similar future widget to submit further draw commands to the GPU (via `wgpu` in the current case) without the risk of a race condition with the normal druid back-end. This mechanism is a workaround for the lack of explicit synchronization or unified rendering pipeline. - Implement the `HasRawWindowHandle` trait for `WindowHandle` to allow native interop to any crate supporting that trait, without explicit dependency to said crate. The new `WgpuView` widget uses those new features to provide a framework to render arbitrary 3D content inside that widget via a new `WgpuRenderer` trait. This widget is only enabled with the new `wgpu_view` feature, because it requires some additional dependencies that users might not be interested in always pulling into their project if not using that widget. A new `wgpu_view` example demonstrates how to use that new widget by embedding a `WgpuView` widget inside multiple `Split` widgets in a configuration similar to common desktop 3D applications (toolbars on the side and top, and 3D view in the center). The sample reuse the cube sample from the `wgpu-rs` project. Known limitations: - The change is limited to support for the Windows platform, although it lays some architecture that should make it easy to add support for other platforms. - This change only add support for single-depth native child windows, that is platform windows which are a direct child of a top-level window. A platform child window cannot be parented to another platform child window. - Interactive animation of the `WgpuView` content (that is, painting outside the `post_render()` callback, for example at a given framerate) is not addressed nor even explored by this change, and may require future work. It is assumed here that the 3D content is static and rendering is entirely druid-event-based (mouse click, window resize, etc.). Bug: linebender#1462
Hi,
I am trying to write a small proof of concept of embedding a
wgpu
surface inside a druid app, as a custom widget. I've started druid yesterday and I am really liking this project, but would like to make sure it is possible to render a 3D view inside a desktop app. The typical use case is a 3D viewport with some 2D UI around it (think something like a game editor / Unity3D / UnrealEngine, or any CAD application, or some 3D modeller like Maya or Max).I've started from the
multiwin
example and lightly modified druid forWindowHandle
to implement theHasRawWindowHandle
trait, which allow creating awgpu
instance from it (is that of interest for #891 by the way? the patch is 10 lines (Windows only)). Then I more or less copy-pasted the "cube" example ofwgpu-rs
into themultiwin
example of druid.Now it almost works, but it seems that some background redraw is still active, so I cannot get a stable image. When the mouse moves out and in of the window borders, or other repaint is triggered, I briefly see the 3D cube, but then immediately get back the normal 2D druid UI on top of it.
Does someone has any idea how I could fix this, or could point me in the direction of where widgets are cleared? I don't do that background repaint myself. I've tried with
PaintCtx::paint_with_z_index(9999)
but that doesn't work either.Many thanks for any pointer! :)
The text was updated successfully, but these errors were encountered: