Rendering a local pipewire stream

[wheaney]

Hi, I'm trying to render a pipewire stream, on the same host (never leaves the device), through vulkan, with as little overhead as possible. I was hoping to keep it simple and not re-invent the wheel, and in my investigation I stumbled across this library. I'm wondering if it's overkill to use this since:

1. I'm never leaving the host, I'm just trying to render from a local pipewire node
2. Because of that, the data is never encoded to anything other than raw RGB
3. I'm mainly concerned about unnecessary latency since it's not actually streaming over a network, it has the potential (and necessity) to render very quickly

Can I achieve the above using this, or am I better off writing my own lightweight vulkan renderer (or is there some other library that's better suited)?

[haasn]

Hi,

1. libplacebo is host-only, it does not have networking code and assumes data is already available in the form of CPU buffers or textures.
2. Ingesting raw RGB is fine, and is actually more performant than ingesting YUV because it skips a conversion step.
3. The only avoidable latency would come from using multiple passes, for example if you want to use non-trivial scaling (e.g. lanczos) or HDR tone-mapping

If you aren't doing tone-mapping with dynamic peak detection, there is no GPU->CPU sync inside libplacebo, so it should be as low-latency as the hardware is capable of (*). In particular, the Vulkan interop API is based around the use of Vulkan semaphores, which can avoid host-side synchronization altogether.

It's not exactly clear to me what your requirements are; so I'm not sure if libplacebo is "overkill". But at the very least, the functionality is there if you need it (e.g. if you decide later on that you actually want to output in YUV format, or convert colorspaces, or apply an ICC profile or 3DLUT).

(*) Excluding zero-copy scan-out paths, which libplacebo does not implement. (They are out-of-scope)

[wheaney]

It's not exactly clear to me what your requirements are

Yeah, sorry for that. Honestly it's super simple, I just have a pipewire stream that I want to render as video, but it has to be through vulkan as I'm trying to apply an implicit layer for some custom shaders (all that is already written, so this is an attempt at reusing it for video). So the only requirement is pipewire (or gstreamer with pipewiresrc if that's simpler) -> vulkan window; being able to fullscreen it borderless is basically it. Implicitly, I also need it to be fast, in case the pipewire stream has game content or something that requires low-latency video.