Linux VAAPI Doesn't Work With AppImage or Flatpak

[RickAndTired]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the nightly release?

• This issue is present in the nightly release

Describe the Bug

Encoder VAAPI fails when using AppImage or the official Flatpak (not from flathub)

Expected Behavior

No response

Additional Context

VAAPI works when using the Sunshine deb

Host Operating System

Linux

Operating System Version

Ubuntu 23.10

Architecture

64 bit

Sunshine commit or version

0.23.0

Package

Linux - AppImage

GPU Type

AMD

GPU Model

RX 6600XT

GPU Driver/Mesa Version

23.2.1

Capture Method (Linux Only)

X11

Config

output_name = 1
adapter_name = /dev/dri/renderD128

Apps

No response

Relevant log output

AppImage:
[2024:04:11:12:18:51]: Info: Sunshine version: 0.23.0
[2024:04:11:12:18:51]: Error: Couldn't load cuda: -1
[2024:04:11:12:18:51]: Error: Failed to gain CAP_SYS_ADMIN
[2024:04:11:12:18:51]: Info: /dev/dri/card0 -> amdgpu
[2024:04:11:12:18:51]: Error: Failed to gain CAP_SYS_ADMIN
[2024:04:11:12:18:51]: Error: Couldn't get handle for DRM Framebuffer [126]: Probably not permitted
[2024:04:11:12:18:51]: Error: You must run [sudo setcap cap_sys_admin+p $(readlink -f sunshine)] for KMS display capture to work!
[2024:04:11:12:18:51]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2024:04:11:12:18:51]: Info: Detecting monitors
[2024:04:11:12:18:51]: Info: Detected monitor 0: DisplayPort-0, connected: true
[2024:04:11:12:18:51]: Info: Detected monitor 1: DisplayPort-1, connected: false
[2024:04:11:12:18:51]: Info: Detected monitor 2: DisplayPort-2, connected: false
[2024:04:11:12:18:51]: Info: Detected monitor 3: HDMI-A-0, connected: true
[2024:04:11:12:18:51]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024:04:11:12:18:51]: Info: Trying encoder [nvenc]
[2024:04:11:12:18:51]: Info: Screencasting with X11
[2024:04:11:12:18:51]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:18:51]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:18:51]: Error: Couldn't load cuda: -1
[2024:04:11:12:18:51]: Error: Couldn't load cuda: -1
[2024:04:11:12:18:51]: Info: Encoder [nvenc] failed
[2024:04:11:12:18:51]: Info: Trying encoder [vaapi]
[2024:04:11:12:18:51]: Info: Screencasting with X11
[2024:04:11:12:18:51]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:18:51]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:18:51]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:18:51]: Info: Color depth: 8-bit
[2024:04:11:12:18:51]: Info: Color range: [JPEG]
[2024:04:11:12:18:51]: Info: System tray created
[2024:04:11:12:18:51]: Info: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function __vaDriverInit_1_0
[2024:04:11:12:18:51]: Error: Couldn't initialize va display: unknown libva error
[2024:04:11:12:18:51]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:18:51]: Info: Color depth: 8-bit
[2024:04:11:12:18:51]: Info: Color range: [JPEG]
[2024:04:11:12:18:51]: Info: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function __vaDriverInit_1_0
[2024:04:11:12:18:51]: Error: Couldn't initialize va display: unknown libva error
[2024:04:11:12:18:51]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:18:51]: Info: Color depth: 8-bit
[2024:04:11:12:18:51]: Info: Color range: [JPEG]
[2024:04:11:12:18:51]: Info: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function __vaDriverInit_1_0
[2024:04:11:12:18:51]: Error: Couldn't initialize va display: unknown libva error
[2024:04:11:12:18:51]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:18:51]: Info: Color depth: 8-bit
[2024:04:11:12:18:51]: Info: Color range: [JPEG]
[2024:04:11:12:18:51]: Info: /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so has no function __vaDriverInit_1_0
[2024:04:11:12:18:51]: Error: Couldn't initialize va display: unknown libva error
[2024:04:11:12:18:51]: Info: Encoder [vaapi] failed
[2024:04:11:12:18:51]: Info: Trying encoder [software]
[2024:04:11:12:18:51]: Info: Screencasting with X11
[2024:04:11:12:18:51]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:18:51]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:18:51]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:18:51]: Info: Color depth: 8-bit
[2024:04:11:12:18:51]: Info: Color range: [JPEG]
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] profile High, level 4.2, 4:2:0, 8-bit
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] frame I:1     Avg QP:31.00  size:  1203
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] mb I  I16..4: 99.9%  0.0%  0.0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] 8x8 transform intra:0.0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] i16 v,h,dc,p: 97%  0%  3%  0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 75% 12%  0%  0%  0%  0% 12%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 100%  0%  0%  0%  0%  0%  0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] i8c dc,h,v,p: 100%  0%  0%  0%
[2024:04:11:12:18:51]: Info: [libx264 @ 0x634d40a6bd40] kb/s:577.44
[2024:04:11:12:18:51]: Info: Screencasting with X11
[2024:04:11:12:18:51]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:18:51]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:18:51]: Info: 
[2024:04:11:12:18:51]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:04:11:12:18:51]: Info: 
[2024:04:11:12:18:51]: Info: Found H.264 encoder: libx264 [software]

Flatpak:
[2024:04:11:12:30:08]: Info: Sunshine version: 0.23.0
[2024:04:11:12:30:08]: Error: Couldn't load cuda: -1
[2024:04:11:12:30:08]: Error: Failed to gain CAP_SYS_ADMIN
[2024:04:11:12:30:08]: Info: /dev/dri/card0 -> amdgpu
[2024:04:11:12:30:08]: Error: Failed to gain CAP_SYS_ADMIN
[2024:04:11:12:30:08]: Error: Couldn't get handle for DRM Framebuffer [132]: Probably not permitted
[2024:04:11:12:30:08]: Error: You must run [sudo setcap cap_sys_admin+p $(readlink -f sunshine)] for KMS display capture to work!
[2024:04:11:12:30:08]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2024:04:11:12:30:08]: Info: Detecting monitors
[2024:04:11:12:30:08]: Info: Detected monitor 0: DisplayPort-0, connected: true
[2024:04:11:12:30:08]: Info: Detected monitor 1: DisplayPort-1, connected: false
[2024:04:11:12:30:08]: Info: Detected monitor 2: DisplayPort-2, connected: false
[2024:04:11:12:30:08]: Info: Detected monitor 3: HDMI-A-0, connected: true
[2024:04:11:12:30:08]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024:04:11:12:30:08]: Info: Trying encoder [nvenc]
[2024:04:11:12:30:08]: Info: Screencasting with X11
[2024:04:11:12:30:08]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:30:08]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:30:08]: Error: Couldn't load cuda: -1
[2024:04:11:12:30:08]: Error: Couldn't load cuda: -1
[2024:04:11:12:30:08]: Info: Encoder [nvenc] failed
[2024:04:11:12:30:08]: Info: Trying encoder [vaapi]
[2024:04:11:12:30:08]: Info: Screencasting with X11
[2024:04:11:12:30:08]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:30:08]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:30:08]: Error: Couldn't create GBM device: [No such file or directory]
[2024:04:11:12:30:08]: Info: System tray created
[2024:04:11:12:30:08]: Error: Couldn't create GBM device: [No such file or directory]
[2024:04:11:12:30:08]: Error: Couldn't create GBM device: [No such file or directory]
[2024:04:11:12:30:08]: Error: Couldn't create GBM device: [No such file or directory]
[2024:04:11:12:30:08]: Info: Encoder [vaapi] failed
[2024:04:11:12:30:08]: Info: Trying encoder [software]
[2024:04:11:12:30:08]: Info: Screencasting with X11
[2024:04:11:12:30:08]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:30:08]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:30:08]: Info: SDR color coding [Rec. 601]
[2024:04:11:12:30:08]: Info: Color depth: 8-bit
[2024:04:11:12:30:08]: Info: Color range: [JPEG]
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] profile High, level 4.2, 4:2:0, 8-bit
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] frame I:1     Avg QP:31.00  size:  1203
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] mb I  I16..4: 99.9%  0.0%  0.0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] 8x8 transform intra:0.0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] i16 v,h,dc,p: 97%  0%  3%  0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 75% 12%  0%  0%  0%  0% 12%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  0%  0% 100%  0%  0%  0%  0%  0%  0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] i8c dc,h,v,p: 100%  0%  0%  0%
[2024:04:11:12:30:08]: Info: [libx264 @ 0x5adf759c5000] kb/s:577.44
[2024:04:11:12:30:08]: Info: Screencasting with X11
[2024:04:11:12:30:08]: Info: Configuring selected monitor (1) to stream
[2024:04:11:12:30:08]: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop
[2024:04:11:12:30:08]: Info: 
[2024:04:11:12:30:08]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:04:11:12:30:08]: Info: 
[2024:04:11:12:30:08]: Info: Found H.264 encoder: libx264 [software]

[gschintgen]

This report documents two independent(?) issues that I can both confirm for the latest nightly AppImage on Ubuntu 22.04 with Intel graphics. I can happily provide more logs if needed.

Issue 1. Sunshine AppImage is indeed unable to load VA-API. The error messages for my Intel laptop are similar except that it fails loading iHD_drv_video.so and i965_drv_video.so.

Issue 2. KMS capture is not working due to missing capabilities. The AppImage is unable to set the capabilities. I'm not sure there's a workaround for this (see AppImage/AppImageKit#881), except to just run the AppImage as root e.g. using sudo -E in order to use the regular user's ~/.config/sunshine directory.

[ReenigneArcher]

@gschintgen I'm trying to work around issue 2 in #2300. I haven't tested the code yet, but the idea is to actually copy the binaries out of the AppImage. This has likely been an issue forever, and the custom AppRun probably only ever worked when the AppImage was extracted. sudo -E might be easier though.

For issue 1, we probably need to manually add those libraries to the AppImage. Here's an example of how we did this in the past (for different libraries though). https://github.com/LizardByte/Sunshine/blame/974c4bd4a1d53a4ed09ba0d17add0845551d7c61/.github/workflows/CI.yml#L460

[gschintgen]

but the idea is to actually copy the binaries out of the AppImage.

Hmm, I thought of such an approach too, but then it's even stranger than having an AppImage with an --install command. (I'm not using AppImages that often though; maybe it's more common than I imagine.) It almost becomes a classic installer instead. But if this is what it takes, then so be it.

Since CAP_SYS_ADMIN is almost root anyway I tried chowning the AppImage to root and setting the setuid bit, but for a mysterious reason it then immediately fails to start with open dir error: Permission denied. (no other output)

As for the other issue, bundling hardware drivers seemed slightly unusual at first, but apparently that's quite normal:

 locate i965_drv_video.so
/snap/gnome-3-38-2004/119/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
/snap/gnome-3-38-2004/143/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
/snap/gnome-42-2204/141/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
/snap/gnome-42-2204/176/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
/snap/moonlight/2330/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
/usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so

and

 ls -l /snap/moonlight/2330/usr/lib/x86_64-linux-gnu/dri
total 521619
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 crocus_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 d3d12_dri.so
-rw-r--r--  5 root root 15101240 Jan 24 13:01 d3d12_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 i915_dri.so
-rw-r--r--  1 root root  1764040 Jul 11  2020 i965_drv_video.so
-rw-r--r--  1 root root 32892408 Feb 13  2023 iHD_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 iris_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 kms_swrast_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 nouveau_dri.so
-rw-r--r--  5 root root 15101240 Jan 24 13:01 nouveau_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 r300_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 r600_dri.so
-rw-r--r--  5 root root 15101240 Jan 24 13:01 r600_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 radeonsi_dri.so
-rw-r--r--  5 root root 15101240 Jan 24 13:01 radeonsi_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 swrast_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 virtio_gpu_dri.so
-rw-r--r--  5 root root 15101240 Jan 24 13:01 virtio_gpu_drv_video.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 vmwgfx_dri.so
-rw-r--r-- 13 root root 32612968 Jan 24 13:01 zink_dri.so

I don't have the slightest idea though how self-contained those libraries are... and also why the AppImage is unable to load the system libraries. What I can tell is that those drivers have a versioned Init function:

$ vainfo
libva info: VA-API version 1.14.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_10
libva info: va_openDriver() returns 0

$ nm -D /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so   | grep Init
000000000005f180 T __vaDriverInit_1_10

Sunshine.AppImage seems to look only for __vaDriverInit_1_0.

[gschintgen]

The VAAPI issue seems to be due to some incoherent library versions. It can be "fixed" by removing the bundled libva*.so files.

Here's what I tried (from memory):

mkdir tmpapp
mkdir tmplibs
cd tmpapp
../sunshine.AppImage --appimage-extract
# check that the extracted version is "working" (with broken VAAPI):
cd squashfs-root
./AppRun
# now move libva out of the way:
mv usr/lib/libva*.so ../../tmplibs
# check again, now it should be able to load VAAPI:
./AppRun

It seems to be an all or nothing situation: either don't bundle libva.so file(s) so that the system one is used, which in turn loads the system *_video.so files, or bundle a complete set of Mesa/VAAPI files. Since the AppImage's raison d'être is portability, it's probably best to bundle all VAAPI related files.

[ReenigneArcher]

it's probably best to bundle all VAAPI related files

I'd agree with this

[gschintgen]

I'll file a PR for the AppImage part of the report:

libva info: VA-API version 1.7.0
libva info: Trying to open /tmp/.mount_sunshiq9efzd/usr/lib/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva error: /tmp/.mount_sunshiq9efzd/usr/lib/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /tmp/.mount_sunshiq9efzd/usr/lib/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
[2024:04:16:00:42:30]: Info: 
[2024:04:16:00:42:30]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:04:16:00:42:30]: Info: 
[2024:04:16:00:42:30]: Debug: ------  h264 ------
[2024:04:16:00:42:30]: Debug: PASSED: supported
[2024:04:16:00:42:30]: Debug: REF_FRAMES_RESTRICT: supported
[2024:04:16:00:42:30]: Debug: CBR: supported
[2024:04:16:00:42:30]: Debug: DYNAMIC_RANGE: unsupported
[2024:04:16:00:42:30]: Debug: VUI_PARAMETERS: supported
[2024:04:16:00:42:30]: Debug: -------------------
[2024:04:16:00:42:30]: Info: Found H.264 encoder: h264_vaapi [vaapi]

[gschintgen]

@RickAndTired : Would you mind testing a build of PR #2429? The AppImage can be downloaded here:
https://github.com/LizardByte/Sunshine/actions/runs/8711575720/artifacts/1419787335
(You must be logged in; don't use wget from the CLI...)

[RickAndTired]

@RickAndTired : Would you mind testing a build of PR #2429? The AppImage can be downloaded here: https://github.com/LizardByte/Sunshine/actions/runs/8711575720/artifacts/1419787335 (You must be logged in; don't use wget from the CLI...)

VAAPI didn't work with this build, here's the log:

sunshine.log

[gschintgen]

Thank you for testing and providing the log!
The relevant part seems to be here:

[2024:04:17:11:44:01]: Info: dlopen of /tmp/.mount_sunshigiryh4/usr/lib/radeonsi_drv_video.so failed: /tmp/.mount_sunshigiryh4/usr/lib/radeonsi_drv_video.so: undefined symbol: amdgpu_va_get_start_addr
[2024:04:17:11:44:01]: Error: Couldn't initialize va display: unknown libva error

There are a number of very recent reports of Mesa breakage referring to that same symbol: https://www.google.com/search?q=%22amdgpu_va_get_start_addr%22
That amdgpu_va_get_start_addr symbol seems to be rather recent: https://gitlab.freedesktop.org/mesa/drm/-/merge_requests/338. It seems to refer to the amdgpu kernel driver.

As I'm now noticing the PR is not working either on my own AMD system (RX6650XT), but with different errors. (While preparing the PR I was using an Intel iGPU-based laptop and it fixed the issue just fine, so I was cautiously optimistic about it.)

The reason that I get past the amdgpu_va_get_start_addr issue might well be that I'm using a PPA with newer mainline kernel images.

I'll see how it goes if I'm adding the stable kisak mesa PPA instead of the "fresh"er variant. (I thought the more bleeding edge PPA might be advantageous for users with more recent hardware, i.e. RX7xxx.)

For completeness, here's the output on my system:

libva info: VA-API version 1.14.0
libva info: Trying to open /tmp/.mount_sunshiVQNzQT/usr/lib/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
amd: LLVM doesn't support , bailing out...
[2024:04:17:21:45:43]: Debug: EGL: [Mesa Project]: version [1.5]
[2024:04:17:21:45:43]: Debug: API's supported: [OpenGL OpenGL_ES ]
[2024:04:17:21:45:43]: Debug: GL: vendor: Mesa
[2024:04:17:21:45:43]: Debug: GL: renderer: llvmpipe (LLVM 15.0.7, 256 bits)
[2024:04:17:21:45:43]: Debug: GL: version: 4.5 (Compatibility Profile) Mesa 23.3.6 - kisak-mesa PPA
[2024:04:17:21:45:43]: Debug: GL: shader: 4.50
[2024:04:17:21:45:43]: Info: SDR color coding [Rec. 601]
[2024:04:17:21:45:43]: Info: Color depth: 8-bit
[2024:04:17:21:45:43]: Info: Color range: [JPEG]
libva info: VA-API version 1.14.0
libva info: Trying to open /tmp/.mount_sunshiVQNzQT/usr/lib/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
[2024:04:17:21:45:43]: Debug: vaapi vendor: Mesa Gallium driver 24.0.5 - kisak-mesa PPA for AMD Radeon RX 6650 XT (radeonsi, navi23, LLVM 15.0.7, DRM 3.57, 6.8.4-zabbly+)
[2024:04:17:21:45:43]: Debug: [AVHWDeviceContext @ 0x563d0a6c0240] VAAPI driver: Mesa Gallium driver 24.0.5 - kisak-mesa PPA for AMD Radeon RX 6650 XT (radeonsi, navi23, LLVM 15.0.7, DRM 3.57, 6.8.4-zabbly+).
[2024:04:17:21:45:43]: Debug: [AVHWDeviceContext @ 0x563d0a6c0240] Driver not found in known nonstandard list, using standard behaviour.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a4e1f80] Input surface format is nv12.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a4e1f80] Using VAAPI profile VAProfileH264High (7).
[2024:04:17:21:45:43]: Error: [h264_vaapi @ 0x563d0a4e1f80] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2024:04:17:21:45:43]: Info: Retrying with fallback configuration options for [h264_vaapi] after error: Function not implemented
libva info: VA-API version 1.14.0
libva info: Trying to open /tmp/.mount_sunshiVQNzQT/usr/lib/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_14
libva info: va_openDriver() returns 0
[2024:04:17:21:45:43]: Debug: vaapi vendor: Mesa Gallium driver 24.0.5 - kisak-mesa PPA for AMD Radeon RX 6650 XT (radeonsi, navi23, LLVM 15.0.7, DRM 3.57, 6.8.4-zabbly+)
[2024:04:17:21:45:43]: Debug: [AVHWDeviceContext @ 0x563d0a8addc0] VAAPI driver: Mesa Gallium driver 24.0.5 - kisak-mesa PPA for AMD Radeon RX 6650 XT (radeonsi, navi23, LLVM 15.0.7, DRM 3.57, 6.8.4-zabbly+).
[2024:04:17:21:45:43]: Debug: [AVHWDeviceContext @ 0x563d0a8addc0] Driver not found in known nonstandard list, using standard behaviour.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Input surface format is nv12.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Using VAAPI profile VAProfileH264High (7).
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Using VAAPI render target format YUV420 (0x1).
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] RC mode: CBR.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] RC target: 100% of 1000000 bps over 1000 ms.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] RC buffer: 1000000 bits, initial fullness 750000 bits.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] RC framerate: 60/1 (60.00 fps).
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Driver does not report any additional prediction constraints.
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Using intra and P-frames (supported references: 1 / 1).
[2024:04:17:21:45:43]: Warning: [h264_vaapi @ 0x563d0a6f1d80] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:04:17:21:45:43]: Debug: [h264_vaapi @ 0x563d0a6f1d80] Using level 4.2.
/tmp/.mount_sunshiVQNzQT/AppRun.wrapped: line 92: 76507 Segmentation fault      (core dumped) LIBVA_DRIVERS_PATH="$HERE/usr/lib" "$SUNSHINE_BIN_HERE" $@

If I have a build that's working on my AMD box I'll let you know.

[gschintgen]

I downgraded the mesa PPA in the AppImage. Now it seems to be working fine on my AMD system. (Logs look good, basic desktop streaming as perfect as it should be.)
I'll post the link as soon as there's a new build available.

(I tested it with the build in my own fork: https://github.com/gschintgen/Sunshine/actions/runs/8728427369/artifacts/1423643042 )

[gschintgen]

https://github.com/LizardByte/Sunshine/actions/runs/8728292196/artifacts/1423740737

[RickAndTired]

https://github.com/LizardByte/Sunshine/actions/runs/8728292196/artifacts/1423740737

This one crashes

sunshine.log

[gschintgen]

Bummer.
It might be related to linuxdeploy's blacklist, notably https://github.com/AppImageCommunity/pkg2appimage/blob/ac4d7b73bd98be7d14644775e9058d09109b2fa5/excludelist#L55.
But the libdrm on your Ubuntu 23.10 is not that old either.
At this point I can only guess. All the more reason to just bundle everything.