[NVIDIA/DRM] Scan-out corruption at every 4K mode, clean at 1080p/1440p; wlroots unaffected on identical DRM state

[moimart]

Summary

On the embedded/DRM backend with an NVIDIA GPU, gamescope shows persistent
scan-out corruption at every 4K mode (3840x2160 @ 30/60/120). The same
machine and panel are completely clean at 1920x1080@120 and 2560x1440@120,
and clean at 3840x2160@120 under wlroots (Hyprland) with a byte-identical
scan-out surface.

It is intermittent per session start (roughly 50%), which makes single-boot
testing unreliable - please treat any "fix" below as needing several boots.

Environment

• gamescope 3.16.25.r7.g6644cc9 (commit 6644cc9), also reproduced on stock
  packaged builds
• NVIDIA RTX 4080 SUPER, driver 610.57.04 (open kernel module)
• LG TV over HDMI 2.1 (FRL), EDID VIC 118 = 3840x2160@119.88
• Kernel 7.1.6, Arch/CachyOS
• Session: gamescope-session-cachyos 1.1.6, Steam gamepadui

Reproduced on a second machine

Also reproduced on a different host with an RTX 4070 driving a different
4K panel (Xiaomi/FireTV, HDMI 2.0, gamescope at 3840x2160@60). Same
gamescope-session-cachyos setup. So this is not specific to one GPU model,
one panel, or one HDMI generation.

Symptom

Vertical column/band corruption showing displaced framebuffer content, at first
confined to the bottom of the screen, later across more of it. Independent of
input, present with static content, present with and without
--force-composition.

The key evidence: it tracks RESOLUTION, not bandwidth

Mode	Pixel clock	Result
1920x1080@120	296700 kHz	clean
2560x1440@120	497750 kHz	clean
3840x2160@30	296700 kHz	broken
3840x2160@60	594000 kHz	broken
3840x2160@120	1186810 kHz	broken

A higher pixel clock is clean while a lower one is broken, which rules out
link bandwidth, DSC, FRL negotiation and cable entirely.

wlroots is clean with an identical surface

Full /sys/kernel/debug/dri/N/state dumps under both compositors at
3840x2160@120 differ in exactly one field - the backing allocation size:

Hyprland (clean):    format=XB30 modifier=0x300000000606014 pitch=15360
                     size=3840x2160  obj size=33423360  (= 15360 * 2176)
gamescope (broken):  format=XB30 modifier=0x300000000606014 pitch=15360
                     size=3840x2160  obj size=33554432  (32 MiB)

Same format, same NVIDIA block-linear modifier, same pitch, same dimensions,
same crtc-pos/src-pos/rotation/zpos, same colour config. Connector state is
also identical (Colorspace = BT2020_RGB in both, empty HDR_OUTPUT_METADATA).

Note this NVIDIA connector exposes very few properties - EDID, DPMS, TILE,
dithering mode, HDR_OUTPUT_METADATA, Colorspace, vrr_capable. No "content type",
no "Broadcast RGB", no "max bpc" - so most of gamescope's property-setting code
is inert here and the two compositors necessarily issue near-identical commits.

Eliminated (each tested on hardware)

• Refresh rate / bandwidth / DSC (see table above)
• The fractional 119.88 Hz timing; --generate-drm-mode fixed (removing it
  produced the same clock and was worse)
• Cursor compositing and --hide-cursor-delay
• --force-composition on and off
• GPU memory-clock starvation (locked to 11501 MHz / P0 - unchanged)
• The vblank redzone (had to patch it in: the GAMESCOPE_TUNEABLE_VBLANK_*
  atom handlers are behind #if 0, see note below)
• Forcing DRM_FORMAT_MOD_LINEAR for flippable images - made it worse
• 8-bit output (XRGB8888 instead of XBGR2101010) - still corrupt
• gamescope's 2x upscale of Steam's 1080p UI (pinned Xwayland to native)
• Block-linear partial tail row (2160 = 16.875 * 128): padding the VkImage
  allocation to whole block rows survived 3 cold boots then reproduced.
  Also note 1080 = 8.4375 * 128 is also partial and is clean.
• Compute imageStore into the exported DMA-BUF: patched gamescope to
  composite into a private optimally-tiled image and vkCmdCopyImage into the
  scan-out buffer (verified active via logging) - still corrupt.
• Display-manager interaction: with plasmalogin (KWin greeter) disabled and
  gamescope launched straight from a TTY, 5 consecutive boots were clean, then
  it reproduced. Forcing a real off->on modeset bounce in a separate atomic
  commit before the modeset also worked once, then failed.

Possibly related

#1964 (embedded gamescope, NVIDIA DRM backend artifacts) - but that is
overlay-triggered; this is present with static content and no overlay.
Also resembles the unfixed NVIDIA forum report at
https://forums.developer.nvidia.com/t/295314 , though that one requires HDR
and this reproduces with HDR off.

---

Two unrelated bugs found while investigating

1. External-display redzone scaling is a no-op

src/vblankmanager.cpp:118

const uint64_t ulRedZone = eScreenType == GAMESCOPE_SCREEN_TYPE_INTERNAL
    ? m_ulVBlankDrawBufferRedZone
    : std::min<uint64_t>( m_ulVBlankDrawBufferRedZone,
          ( m_ulVBlankDrawBufferRedZone * 60'000 * nRefreshRate ) / 60'000 );

The 60'000 factors cancel, leaving redzone * nRefreshRate, which for any
refresh rate >= 1 is >= the unscaled value - so std::min always returns the
unscaled m_ulVBlankDrawBufferRedZone. The refresh-rate scaling described in
the comment above it ("scale it by our target refresh so we don't miss
submitting for vblank in DRM ... fixes wonky frame-pacing on 4K@30Hz screens")
never executes on any external display.

2. The vblank tunables are dead

GAMESCOPE_TUNEABLE_VBLANK_REDZONE and
GAMESCOPE_TUNEABLE_VBLANK_RATE_OF_DECAY_PERCENTAGE are interned and
documented, but their handler is inside #if 0 (steamcompmgr.cpp:6208-6217),
and the global it writes (g_uVblankDrawBufferRedZoneNS) is read nowhere in
the tree. Setting either property silently does nothing.

[Forge64]

Also affected by this, RTX 5090 connected via Displayport to a 4K/160Hz display. I am able to reproduce on every gamescope session, however. I cannot get a clean screen at any time.

The corruption appears to have structure, and it varies from startup to startup. Today I could what looked like a scanout buffer in the bottom right of the screen with what looked like mangohud elements being drawn/redrawn there, but enabling the performance overlay in Steam didn't affect it.

[SimpleHeuristics]

This was better on drivers between 575 and 595. The 610 series seems to have a regression to pre 575 behavior.

I don't think Nvidia has acknowledged this issue in 610. I unfortunately don't know enough to say if this is a Gamescope issue, steam client issue, or truly an Nvidia side issue which most people tend to try and blame on.

The only thing I can add to this though is if you are in Gamescope session and launch a native HDR enabled game like cyberpunk or assassins creed shadows or spider man 2 etc it seems like for some reason that switching of the color space (rather than having Gamescope do ITM of SDR content) gets rid of the visual corruption until you bring up any part of the steam UI or the performance overlay

If you just use an SDR game then the visual artifacts remain even with overlay or UI not in view.