x11 with Xwayland: requesting fullscreen causes BadRRCrtc and makes subsequent SDL programs fail

[smcv]

I'm trying to upgrade the SDL package in Debian from 2.0.14 to the recent 2.0.16 release, and encountered a regression. I'm running GNOME 3.38 in Wayland mode with Xwayland, which will be the default desktop environment in Debian 11 (we can't have GNOME 40 in Debian 11, because that wasn't released until after Debian 11 had already frozen).

When I run a fullscreen SDL program like testgl2 --fullscreen or OpenArena using the default X11 video backend for SDL, the updated SDL fails with an X11 error:

X Error of failed request:  BadRRCrtc (invalid Crtc parameter)
  Major opcode of failed request:  140 (RANDR)
  Minor opcode of failed request:  20 (RRGetCrtcInfo)
  Crtc id in failed request: 0x0
  Serial number of failed request:  243
  Current serial number in output stream:  243

SDL 2.0.16 triggers this, but a modified SDL 2.0.16 with b033cd0, 4c7825f, d0effad and 16e3bfe reverted does not trigger it.

When this happens, it changes the state of my Xwayland display, leaving the mode unset.

After the state has changed, running any other graphical SDL program with the X11 backend (for example testgl2, or openarena again) fails, this time with a different error:

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  151 (XFree86-VidModeExtension)
  Minor opcode of failed request:  1 (XF86VidModeGetModeLine)
  Value in failed request:  0x522
  Serial number of failed request:  63
  Current serial number in output stream:  63

After the state has changed, all versions of SDL that I have tried will fail in the same way (I tried vanilla 2.0.16, 2.0.16 + reverts as listed above, and 2.0.14dfsg2-3 from Debian 11), which means it's easy to get the wrong idea about which versions work and which versions don't.

The changed state persists until I log out from GNOME and back in.

Original report (misleading)

Games like openarena and darkplaces, and also simpler test programs like testgl2 and testvulkan, fail with:

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  151 (XFree86-VidModeExtension)
  Minor opcode of failed request:  1 (XF86VidModeGetModeLine)
  Value in failed request:  0x522
  Serial number of failed request:  63
  Current serial number in output stream:  63

Workaround: if I run with SDL_VIDEODRIVER=wayland, the games and test programs work (but without window decorations because we don't have libdecor available yet - I'm working on that in parallel).

This seems like might be related to #4561, where @icculus mentioned that XVidMode "doesn't work in SDL2 in almost all cases", but the failure mode is different. I have not done anything special to force Xrandr to be disabled, and I don't have any SDL_* in my environment.

I'm building with essentially the same options that we used in the 2.0.14 package, and in particular configure.ac is successfully detecting Xrandr.

I tried building 2.0.16 with b033cd0, 4c7825f, d0effad and 16e3bfe reverted, but that doesn't seem to resolve this.

[icculus]

Can you post config.log here?

[smcv]

Actually, this seems to be a regression somewhere else on the system, rather than a SDL regression: I downgraded to the Debian 11 package (2.0.14+dfsg2-3) and I'm still getting this error. It worked when I uploaded 2.0.14+dfsg2-3 back in January, so there must be something else going wrong.

[smcv]

I can't reproduce this after logging out from GNOME and back in, either with 2.0.14 or with 2.0.16.

[smcv]

Ugh, this might depend on previous state.

On a fresh login to GNOME, I have:

$ xrandr 
Screen 0: minimum 16 x 16, current 1920 x 1080, maximum 32767 x 32767
XWAYLAND0 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 310mm x 170mm
   1920x1080     59.96*+

and I can run testgl2 and testvulkan successfully.

On one machine (a laptop), when I run openarena, configured for full-screen, I get a different error:

ioq3 1.36+u20201117.d1b7ab6~dfsg-1/Debian linux-x86_64 Dec 14 2020
SSE instruction set enabled
----- FS_Startup -----
We are looking in the current search path:
/home/smcv/.openarena/baseoa
/usr/lib/openarena/baseoa
/usr/lib/openarena/baseoa/z_oacmp-volume1-v3.pk3 (370 files)
/usr/lib/openarena/baseoa/pak6-patch088.pk3 (711 files)
/usr/lib/openarena/baseoa/pak6-patch085.pk3 (559 files)
/usr/lib/openarena/baseoa/pak6-misc.pk3 (229 files)
/usr/lib/openarena/baseoa/pak5-TA.pk3 (139 files)
/usr/lib/openarena/baseoa/pak4-textures.pk3 (1753 files)
/usr/lib/openarena/baseoa/pak2-players.pk3 (669 files)
/usr/lib/openarena/baseoa/pak2-players-mature.pk3 (231 files)
/usr/lib/openarena/baseoa/pak1-maps.pk3 (100 files)
/usr/lib/openarena/baseoa/pak0.pk3 (1042 files)
    
----------------------
5803 files in pk3 files
execing default.cfg
execing q3config.cfg
execing autoexec.cfg
Hunk_Clear: reset the hunk ok
----- Client Initialization -----
----- Initializing Renderer ----
Trying to load "renderer_opengl2_x86_64.so" from "/usr/lib/ioquake3"...
-------------------------------
QKEY found.
----- Client Initialization Complete -----
----- R_Init -----
SDL using driver "x11"
Initializing OpenGL display
Display aspect: 1.778
...setting mode -2: 1920 1080
tty]X Error of failed request:  BadRRCrtc (invalid Crtc parameter)
  Major opcode of failed request:  140 (RANDR)
  Minor opcode of failed request:  20 (RRGetCrtcInfo)
  Crtc id in failed request: 0x0
  Serial number of failed request:  243
  Current serial number in output stream:  243

(But on another machine, a desktop with an external display, it works fine.)

After the RRGetCrtcInfo error, the xrandr state has changed: the mode of the XWAYLAND0 output has been unset:

Screen 0: minimum 16 x 16, current 1920 x 1080, maximum 32767 x 32767
XWAYLAND0 connected primary (normal left inverted right x axis y axis)
   1920x1080     59.96 +

and subsequent runs of both testgl2 and openarena start failing:

$ /usr/lib/x86_64-linux-gnu/installed-tests/SDL2/testgl2   
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  151 (XFree86-VidModeExtension)
  Minor opcode of failed request:  1 (XF86VidModeGetModeLine)
  Value in failed request:  0x522
  Serial number of failed request:  63
  Current serial number in output stream:  63

[smcv]

A more verbose version of the state change: before running openarena

$ xrandr --verbose
Screen 0: minimum 16 x 16, current 1920 x 1080, maximum 32767 x 32767
XWAYLAND0 connected primary 1920x1080+0+0 (0x22) normal (normal left inverted right x axis y axis) 310mm x 170mm
	Identifier: 0x21
	Timestamp:  233790193
	Subpixel:   unknown
	Gamma:      1.0:1.0:1.0
	Brightness: 0.0
	Clones:    
	CRTC:       0
	CRTCs:      0
	Transform:  1.000000 0.000000 0.000000
	            0.000000 1.000000 0.000000
	            0.000000 0.000000 1.000000
	           filter: 
	non-desktop: 0 
		supported: 0, 1
  1920x1080 (0x22) 173.000MHz -HSync +VSync *current +preferred
        h: width  1920 start 2048 end 2248 total 2576 skew    0 clock  67.16KHz
        v: height 1080 start 1083 end 1088 total 1120           clock  59.96Hz

and after

$ xrandr --verbose
Screen 0: minimum 16 x 16, current 1920 x 1080, maximum 32767 x 32767
XWAYLAND0 connected primary (normal left inverted right x axis y axis)
	Identifier: 0x21
	Timestamp:  234157697
	Subpixel:   unknown
	Clones:    
	CRTCs:      0
	Transform:  1.000000 0.000000 0.000000
	            0.000000 1.000000 0.000000
	            0.000000 0.000000 1.000000
	           filter: 
	non-desktop: 0 
		supported: 0, 1
  1920x1080 (0x22) 173.000MHz -HSync +VSync +preferred
        h: width  1920 start 2048 end 2248 total 2576 skew    0 clock  67.16KHz
        v: height 1080 start 1083 end 1088 total 1120           clock  59.96Hz

[smcv]

I've updated the description to reflect my current understanding of this bug. There are two failure modes, not just one.

The root cause seems to be that when I run openarena with the new SDL, it reconfigures my Xwayland's Xrandr configuration, leaving it in a broken state. I'll try to make a more minimal reproducer for this than all of openarena. This part of the bug only affects 2.0.16, and can be avoided by reverting #4494 and the follow-up commits.

After Xwayland's Xrandr state has been changed like this, all versions of SDL fail to start up - not just 2.0.16, but also older versions. I think this might not be a real bug in those versions: obviously it would be ideal if they could recover, but I haven't been able to work out how to recover from the bad state using xrandr(1) either. This can be avoided by making sure not to get Xwayland into the bad state in the first place.

[smcv]

I can also reproduce this with testgl2 --fullscreen, which is a much simpler reproducer.

[smcv]

After logging out and back in to reset the state, SDL_VIDEO_X11_XRANDR=0 testgl2 --fullscreen also does not trigger this.

[bluestang2006]

this is very similar to the issue I am having with Quake3e, however, I am only experiencing this with the Vulkan renderer in Fullscreen mode only (windowed works)...OpenGL works in both Fullscreen and Windowed...

https://github.com/ec-/Quake3e/issues/119

https://gitlab.freedesktop.org/mesa/mesa/-/issues/5207

Perhaps there is something related?