[X11] Workaround 1x1 Wine window to allow unredirect, fix not working setting responsible for unredirect.

[itz-me-zappex]

Description

Finally, after previous two PRs I have created (and closed) with chunky workarounds and wild activity in #701, I have found solution and figured out that root of issue is bugged rectangles of child windows (1x1+0+0), which are created by OpenGL games running through Wine/Proton (e.g. Geometry Dash) and even native ones running through Steam (e.g. native Terraria). That does not happen with OpenGL games running completely natively (i.e. w/o Steam container) and does not happen with Vulkan games at all as those are have proper rectangles on child windows as xwininfo -tree -root shows, so solution oriented only at that specific case.

Solution

So, the best way I found to solve this, is just not override currently handled window rectangles with new (child) ones if those are bugged (i.e. resolution of child reported as 1x1), that makes muffin use parent window rectangles which are correct.

That "filter" should be used two times:

1. in window.c: to allow FLIP unredirection for such windows by handling those with proper rectangles.
2. in window-x11.c: to avoid unredirection breakage if window mode appears changed e.g. from fullscreen to windowed and back, or when OSD stuff draws over unredirected window.

Also I fixed issue in meta-surface-actor-x11.c related to handling internal unredirect requests if "Disable compositing for full-screen windows" disabled in settings (or through dconf), as those requests always have been handled and fullscreen windows were unredirected.

Closes #701.

[ndiruhniu]

@mtwebster

[itz-me-zappex]

Update on this, I noticed that screen recording software like OBS Studio breaks in-game VSYNC and enforces BLIT rendering until I close screen recorder in case with such buggy windows, I figured out that I also should avoid overriding of window->depth if window has 1x1 size to prevent this issue from happening (and window->xvisual just in case, but that is not real reason of this issue).

[Enokilis]

Would it also be possible to fix the root of the issue?
Why does Proton/Wine create a 1x1 window in the first place? I can actually see it in the corner, manifesting as the desktop's background color.

[itz-me-zappex]

Just merged and summarized commits.

[itz-me-zappex]

As far as I understand, fixing it not planned at all, right? To be more correct - workarounding. And #701 issue is still opened for around 9 months and successfully ignored by development team.

I have found report on Wine's Bugzilla created in 2021
https://bugs.winehq.org/show_bug.cgi?id=50717

And my changes aimed to ignore this 1x1 window, otherwise window will not be unredirected, so that compositor will reduce performance and cause microstuttering on monitor side.

I can confirm that is weird that native games from Steam also have the same issue. And again, I will add that issue is not reproducible with games using Vulkan API (including those running with DXVK or VKD3D), only OpenGL stuff suffers from that.

In case someone uses Arch Linux like me, you can build and install my fork:

# Official PKGBUILD from Arch Linux GitLab
git clone https://gitlab.archlinux.org/archlinux/packaging/packages/muffin

# Go to directory with PKGBUILD
cd muffin/

# Install build dependencies, download and extract source code 
makepkg --syncdeps --nobuild

# Replace extracted source code with one from my fork
rm -rf src/muffin-6.4.1
git clone https://github.com/itz-me-zappex/muffin-fixed-unredirection.git src/muffin-6.4.1

# Build and install
makepkg -i

Or just modify PKGBUILD by skipping checksums and replacing source with one from my repository.

After installation press Ctrl+Alt+Esc to apply changes.

[Enokilis]

Followed the steps, but I still see a visible 1x1 window in the corner and get no flipping.

[itz-me-zappex]

Followed the steps, but I still see a visible 1x1 window in the corner and get no flipping.

1x1 window will not disappear.

About flipping, make sure in that you have "Allow flipping" and "Show API indicator" enabled in nvidia-settings app.

The most important: open system settings, then go to General and enable unredirection of fullscreen windows, now this switcher works as it should (in vanilla code this setting appears ignored so that windows appear unredirected despite setting turned on or off).