v3 rewrite

[m1k1o]

Since Vue2 is deprecated #358 we need to rewrite client. While we will be rewriting client, we could take a look at the server as well and finally join m1k1o/neko and demodesk/neko.

Main pain-points that should be solved by this rewrite:

• Client should be split to library typescript component that does not use vuejs or any library and can be imported by any project. It should be as easy to integrate into custom project as embedding video player. Similar to how demodesk/neko-client is built, but without VueJs.
• The connection, media streaming and control should be extracted as an interface, so that it can be implemented by various protocols, not just WebSockets+WebRTC. That would elevate this project from just a shared virtual environment to basically video streaming server with built in tools for feedback and out-of-band communication (such as natively binding to RDP/VNC protocols, controlling drones/robots/PTZ cameras/industrial devices remotely). Since the controlling layer could be just a plugin, it does not need to relay on only keyboard and mouse but would allow to plug in gamepads, joysticks or even Virtual Reality glasses (anything).

Connection:

Neko can connect to the backend using multiple channels. Therefore API users should not be exposed to WebSocket internals.

They should only care about the connection status:

• 'connected' - user is connected to the server
• 'connecting' - currently the client is attempting to establish connection to the server.
• 'disconnected' - user is disconnected from the server and there are no attempts to connect to the server. this should always be notified with a reason why it has been disconnected.

And about connection type:

• 'none' - no connection is currently used.
• 'short_polling' - every X ms the client requests the server for updates.
• 'long_polling' - http request is kept open until the server has updates to send to the client. then the client sends another request.
• 'sse' - server sends updates to the client using Server-Sent Events.
• 'websocket' - server sends updates to the client using WebSockets.
• ... others (e.g. mqtt...)

Media streaming

For media streaming, we implement a similar approach with the following streaming backends:

• 'none' - no media streaming is currently streamed.
• 'm3u8' - media is streamed using HLS.
• 'webrtc' - media is streamed using WebRTC.
• 'quic' - media is streamed using QUIC.
• ... others (e.g. rtsp, dash...)

Various media streaming backends can have various features. For example, WebRTC can have a feature to send media to the server, while HTTP can only receive media from the server.
They can be selected based on the user's device capabilities, network conditions, and server capabilities.
There must be a single interface that all streaming backends must satisfy and its their only communication channel with the rest of the system.

Control (Human interface device)

The user can control the target system using various human interface devices. The user can use a keyboard, mouse, gamepad, touch screen, or any other device that can be used to control the system. Custom or virtual devices can be used as well.

Normally in-band feedback should be provided to the user inside media stream. But there can be cases where out-of-band feedback is required.

• when the user is using a gamepad, in can be used to control the system, but the gamepad can also have a vibration feature that can be used to provide feedback to the user.
• cursors can be hidden on the screen, or for certain users therefore they need to be transmitted out-of-band.
• since there are multiple users all of them can have custom cursor shown on the screen, but only one cursor can be controlled by the user. Therefore the cursor position must be transmitted out-of-band.
• changing screen resolution, orientation, or other settings.
• changing the keyboard layout or modifier keys.
• setting a host - who is currently controlling the system based on priority and permissions.

Control can use both underlying connections or media streaming for transmitting and receiving control data. For example, WebRTC data channels can be used for transmitting control data in real-time.

Conclusion

In the first step we should create and agree on client's library API and create interfaces that will be used and implemented later.

More will follow, please let me know if you have any ideas. Stay tuned.

Todo

First phase - merge demodesk/neko to m1k1o/neko with legacy driver to emulate old API.

• see more details in Version 3 - Phase 1 #423

Second phase - merge demodesk/neko-client to m1k1o/neko while upgrading to vue3 and deprecate legacy API.

• Rebuild m1k1o/neko GUI from scratch on vue3 while using demodesk/neko-client as core plugin.

Third phase - make the codebase modular (as mentioned above).

[ehfd]

Wayland Support

This would require gstreamer1.0-pipewire or kmsgrab. The latter requires setcap cap_sys_admin+ep, thus might not be practical. Weylus and Rustdesk uses gstreamer1.0-pipewire through xdg-desktop-portal, so this may be preferable.

Implementations readily available at https://github.com/H-M-H/Weylus, https://github.com/pavlobu/deskreen, and of course https://github.com/LizardByte/Sunshine.
More reference:
https://github.com/any1/wayvnc
https://github.com/bbusse/swayvnc-firefox
https://github.com/boppreh/keyboard
https://github.com/boppreh/mouse
https://github.com/maliit/keyboard

Another alternative is https://github.com/games-on-whales/gst-wayland-display by @ABeltramo and @Drakulix. This is by using nested compositors. However, this would likely not work when in conjunction with real monitors.

PipeWire as a hub for all media (Wayland video capture, V4L2 webcam stream dispensation, and PulseAudio drop-in replacement)

Switch to PipeWire for the containers and accept PipeWire audio (for either X11 or Wayland) as well as screen capture (for Wayland) directly as well as PulseAudio (stop-gap solution would be pipewire-pulse). gstreamer1.0-pipewire exists for this.

Moreover, an interesting capability Pipewire has is its potential to replace v4l2loopback with pipewire-v4l2.
It uses LD_PRELOAD (much like the Selkies uinput joystick interposer) to intercept v4l2 communication and route clients to ingest streams from PipeWire. Thus webcams and other video media can utilize this pathway.
This is very interesting as it will work better with containers and Kubernetes and does not require the loopback kernel device.

It's possible to compile the PipeWire GStreamer plugin together with GStreamer if it is included as a subproject in meson.build. Else, use the pipewire-debian PPA.

[ehfd]

I've heard from @totaam that WebSockets for the stream, not only signaling (or any other TCP ways to do it) works until there's packet loss. When there's packet loss, there would be visible defects in the stream.

[ehfd]

QUIC / WebTransport + WebCodecs & WebAudio

For QUIC, it should (obviously) be over HTTP/3 WebTransport instead of a custom QUIC protocol to be compatible with web browsers. Even if we're talking about native clients, HTTP/3 WebTransport would offer no disadvantages.

Note that https://developer.mozilla.org/en-US/docs/Web/API/WebTransport/WebTransport#servercertificatehashes needs to be generated for self-signed certificates at every session start, as the self-signed certificate maximum length is 14 days.

It is worth noting that because WebTransport lacks several capabilities (including the availability of reverse proxies), the technology might need to wait to mature.
w3c/webtransport#525

WebCodecs should be used for decoding in any other protocol than WebRTC, and fall back to WebAssembly + MSE libraries or other different methods if it doesn't exist (since it's pretty new).

It is important to understand that WebSockets should typically not be mixed with HTTP/3 or WebTransport to obtain maximum benefit from HTTP/3. The reason WebSockets were required in WebRTC was to exchange signaling information. HTTP/3 can be used for this very purpose as well and should be.

MediaStream/getUserMedia should still be used for WebRTC. DataChannels cannot transport video or audio efficiently.

All methods for video processing in browsers:
https://webrtchacks.com/video-frame-processing-on-the-web-webassembly-webgpu-webgl-webcodecs-webnn-and-webtransport/
https://rxdb.info/articles/websockets-sse-polling-webrtc-webtransport.html

https://www.youtube.com/watch?v=0RvosCplkCc
https://github.com/colinmarc/magic-mirror
https://github.com/mira-screen-share/sharer
https://github.com/netrisdotme/netris
https://quic.video/
https://developer.mozilla.org/en-US/docs/Web/API/WebTransport/WebTransport
https://www.w3.org/2021/03/media-production-workshop/talks/chris-cunningham-webcodecs-videoencoderconfig.html
https://docs.libp2p.io/concepts/transports/webtransport/
https://github.com/quic-go/webtransport-go

https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API
https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

WebSockets for both media and signaling

Also worth noting is that some restricted firewalled environments need to use WebSockets (with upgrade from HTTP/1.1 required) for both media and signaling. This might be worth one of the options.

For WebSockets, WebCodecs, then falling back to WebAssembly + MSE might work for video and audio decoding.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism#upgrading_to_a_websocket_connection

RTWebSocket is a pretty interesting option for the WebSocket portion of this project.

https://github.com/zenomt/rtwebsocket
https://github.com/zenomt/rtmfp-cpp

https://www.rfc-editor.org/rfc/rfc7016.html

Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7016.html#section-2.3.11.1.2

https://www.rfc-editor.org/rfc/rfc7425.html

Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7425.html#section-5.3.5

Along with WebTransport, the possible reason to use this is to show frames as soon as they arrive instead of going through internal jitterbuffers which WebRTC has limited control over (but the WebRTC approach of having separate audio and video streams should still make the frame display as fast as possible).

Two things to keep in mind about RTWebSocket when looking at the above links:

RTWebSocket uses the same primary abstraction as RTMFP: unidirectional ordered message-oriented "flows" named by arbitrary binary metadata, where each flow can have an independent priority/precedence (that you can change at any time), each message can have an arbitrary transmission deadline (that you can change at any time), and "return flow association" generalizes bidirectional communication into arbitrarily-complex trees of unidirectional flows.
To fully realize the "Real-Time" benefits of RTWebSocket (or RTMFP), you need to avail yourself of the real-time facilities of priority/precedence and transmission deadlines.

Note that you can revise a message's transmission deadline after it's been queued (for example, you can revise the transmission deadlines of previously-queued video messages when you get a new keyframe).

The link to Section 5.3.5 of RFC 7425 above is to an illustration of multiple levels and many more than 2 flows of a bidirectional flow tree.

RFC 7016 describes RTMFP (a UDP-based transport protocol suspiciously similar to, but significantly predating, Quic). RFC 7425 describes how to send RTMP video, audio, data, and RPC messages over RTMFP. the same method can be used to send RTMP messages over RTWebSocket, and the https://github.com/zenomt/rtwebsocket and https://github.com/zenomt/rtmfp-cpp repos have demonstrations of doing that.

WebRTC Improvements

Investigate WHIP and WHEP, allowing unlimited users through WebRTC:
https://bloggeek.me/whip-whep-webrtc-live-streaming/
https://gstreamer.freedesktop.org/documentation/rswebrtc/whipclientsink.html?gi-language=c
https://gstreamer.freedesktop.org/documentation/rswebrtc/whipserversrc.html?gi-language=c
https://gstreamer.freedesktop.org/documentation/webrtchttp/whepsrc.html?gi-language=c
However, WHIP and WHEP don't offer DataChannels, thus requiring another stream or in usage simply for view-only users.

Chunked DataChannel for clipboard and other large information: https://groups.google.com/g/discuss-webrtc/c/f3dfmu3oh00
WebRTC TURN REST API usage: #370
Investigate MultiOpus: node-webrtc/node-webrtc#603 (comment)

Miscellaneous

Developers must understand that all protocols and APIs are bound by web browser specifications and standards. Luckily, we have reached an era where most required APIs for this project objective are supplied by web browsers. In some cases, look for an external JavaScript or WebAssembly implementation.

DASH is available through JavaScript fallback.
https://caniuse.com/mpeg-dash
https://github.com/Dash-Industry-Forum/dash.js

RTSP seems to be not supported directly in the web browser; rather a protocol to capture from another source accessible to the server, than a transport protocol on the web.
https://stackoverflow.com/questions/2245040/how-can-i-display-an-rtsp-video-stream-in-a-web-page

Same for QUIC; QUIC on a web browser is only done with WebTransport.

MQTT is also redundant; all MQTT web client libraries use WebSockets to transport MQTT requests.

[totaam]

When there's packet loss, there would be visible defects in the stream.

It can happen if partial decoding is implemented, but that's very hard to get right.
More likely, you will just get delayed frames, stuttering and bandwidth issues.
Generally, to get visible defects, you need a transport layer that handles packet loss. TCP and websockets do not.

[ehfd]

I included "delayed frames, stuttering and bandwidth issues" into visible (I think I meant perceivable) defects.
But anyways, yes, it's not about artifacts on the video, I agree.

[ehfd]

GPU Acceleration

Zero-copy buffers are very important for shaving off the last bits of latency.

#291 (SW: openh264enc, x265enc, rav1enc, AOM av1enc / NVIDIA nvcudah264enc, nvcudah265enc, nvav1enc - not implemented yet / VA-API vah264enc, vah265enc, vaav1enc)

Performance optimizations: https://git.dec05eba.com/gpu-screen-recorder-gtk/about/ (I encourage talking to the maintainer)

NVIDIA

NVIDIA Capture API (NvFBC) zero-copy framebuffer encoding for NVIDIA GPUs (may lead to great X11 performance improvements): https://github.com/CERIT-SC/gstreamer-nvimagesrc

GStreamer >= 1.22 supports nvcudah264enc and nvcudah265enc. This enables zero-copy encoding pipelines compared to previous nvh264enc and similar plugins. AV1 encoders may be supported later on.

Jetson (aarch64)

https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/SD/Multimedia/AcceleratedGstreamer.html

VA-API (AMD and Intel)

More notes: https://github.com/selkies-project/selkies-gstreamer/blob/332001b5314c352c06f1adb4af3a67b6b7b31fb1/src/selkies_gstreamer/gstwebrtc_app.py#L311

A successful VA-API pipeline which was quite redundant for me so far. vaapih264enc is deprecated and the new plugin everyone is focusing on (since GStreamer 1.22) is vah264enc.
qsvh264enc using Intel QuickSync is also an interesting option and while amfh264enc is not supported in Linux, may be revived with the new Vulkan video encoding APIs.
https://www.khronos.org/blog/khronos-finalizes-vulkan-video-extensions-for-accelerated-h.264-and-h.265-encode

I do not guarantee everything will work for all GPUs, libva version, VA drivers, etc., but I feel it's much better than the deprecated plugins including working well on iHD drivers.

Some more pointers for GPU acceleration and high-performance streaming:
selkies-project/selkies-gstreamer#34
https://github.com/colinmarc/magic-mirror
https://github.com/hgaiser/moonshine
https://github.com/cea-sec/sanzu

Hardware-accelerated JPEG encoding

NVIDIA and VA-API both provide hardware-accelerated JPEG encoding and are supported by most modern GPUs.
This may provide an alternative protocol that goes back to the VNC methodologies, except that it would be hardware-accelerated. JPEG can send each pixel independently, which may prove useful for partial screen refreshes.

https://gstreamer.freedesktop.org/documentation/nvcodec/nvjpegenc.html?gi-language=c

Miscellaneous

Note that x264 requires screen resolutions to be an even number. It wouldn't hurt to default to that always.
selkies-project/selkies-gstreamer#124

GStreamer examples: https://gist.github.com/hum4n0id/2760d987a5a4b68c24256edd9db6b42b

GStreamer Portable Build

GStreamer may be built statically if using C, Rust, or Go (not for Python). Because the most prominent performance and encoding optimizations are in the latest production releases, the most recent releases must be used. Then, Neko can be deployed regardless of environment, standalone without containers.
https://blogs.igalia.com/scerveau/discover-gstreamer-full-static-mode/
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3406

Even if we don't do a static build, it could be useful to make shared library builds for GStreamer which is compatible with all active distros for ABI and glibc (separate GPL and LGPL builds). musl libc is also a viable option especially if there is no Python component.

Using AppImage is also a way to make the resulting application portable. Used in Sunshine and RustDesk.

Conda (https://conda-forge.org/news/2023/07/12/end-of-life-for-centos-6/), used frequently in science and technology, maintains a portable compiler toolchain and package ecosystem (based on CentOS 7) and packages recent GStreamer versions. Especially useful when Python components exist.

[ehfd]

Low-latency graphical streaming (including 3D graphics development - Teradici/NICE DCV and game development) and relative cursors

#339 (Reduce Latency by using eliminating A/V sync for WebRTC or QUIC)

Host Encoder and WebRTC Web Browser Decoder Settings (eliminate all client-side latency)

selkies-project/selkies-gstreamer#34 (comment)
selkies-project/selkies-gstreamer#78
https://multi.app/blog/making-illegible-slow-webrtc-screenshare-legible-and-fast
https://www.rtcbits.com/2023/05/webrtc-header-extensions.html

#344 (Relative cursors in X11 and Wayland, pointer lock)
selkies-project/selkies-gstreamer#107 (Keyboard Lock and Pointer Lock are important. What's also important is the Press and Hold ESC to exit fullscreen capability).

#364 (Unicode Keysyms)

selkies-project/selkies-gstreamer#22 (Are there touch keyboards for mobile users?)

selkies-project/selkies-gstreamer#25 (URL authentication and JSON Web Token authentication?)

selkies-project/selkies-gstreamer#98:
(Handle (1) local scaling - fit client interface to server screen without server display resolution change, and (2) remote scaling - change server display resolution to fit server screen to client interface, and (3) scale cursor movement as well so that it isn't faster or slower than intended by the user)

selkies-project/selkies-gstreamer#102
selkies-project/selkies-gstreamer#99
(Validate cursor behavior with touchpad always, but shouldn't be a big issue since most of our fixes are from Neko itself)

selkies-project/selkies-gstreamer#110 (HiDPI management: https://wiki.archlinux.org/title/HiDPI / https://linuxreviews.org/KDE_Plasma#Workaround_For_Bugs)

And of course, a heap of information in https://docs.lizardbyte.dev/projects/sunshine/en/latest/index.html

More reference:
https://web.archive.org/web/20211003152946/https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc
https://github.com/parsec-cloud/parsec-sdk/blob/master/sdk/parsec.h
https://github.com/evshiron/parsec-web-client
https://github.com/robterrell/web-client
https://github.com/pod-arcade
https://github.com/giongto35/cloud-game
https://github.com/giongto35/cloud-morph

Gamepads/Joysticks and Wayland Input

In the web browser perspective, the interface could utilize whatever the Gamepad API exposes.
https://developer.mozilla.org/en-US/docs/Web/API/Gamepad_API
This is implemented in https://github.com/selkies-project/selkies-gstreamer/blob/main/src/selkies_gstreamer/gamepad.py and https://github.com/selkies-project/selkies-gstreamer/blob/main/addons/gst-web/src/gamepad.js.

But from the server, it is not trivial to use arbitrary input devices in an unprivileged Docker or Kubernetes container, because /dev/uinput is required in most cases. For instance, Sunshine requires /dev/uinput in order to be run.

However, a number of workarounds are available.

https://github.com/selkies-project/selkies-gstreamer/tree/main/addons/js-interposer
This is a LD_PRELOAD method to redirect input calls without using /dev/uinput.

Additional approaches:
https://github.com/JohnCMcDonough/virtual-gamepad
https://github.com/games-on-whales/wolf/tree/stable/src/fake-udev
https://github.com/Steam-Headless/dumb-udev

What's intriguing in this approach is that this workaround method may also pave ways to replace /dev/uinput in Wayland for cursor and keyboard control. X11 included its own interface for input, but Wayland no longer has that and replaced with /dev/uinput. This may allow for a vendor-neutral method to process input in Wayland.

Touchscreen and Stylus

https://github.com/H-M-H/Weylus
https://github.com/pavlobu/deskreen

[ehfd]

Multi-user GPU Sharing

https://dev.to/douglasmakey/a-simple-example-of-using-unix-domain-socket-in-kubernetes-1fga
Looks like it does work now.

• From Discord a few months ago

The issue with this is that this cannot be shared between different Kubernetes pods. It only works within multiple containers within the same pod. This means that GPU sharing with a single X server is a bit harder.

An alternative would be to use X11 through TCP instead of UNIX Sockets.

VirtualGL through GLX or using Wayland would also be an alternative that makes things smoother.

Multi-architecture Environments

Support arm64 (aarch64), arm/v7 (armhf), and ppc64le - for people happening to use the Sierra or Summit supercomputers, perhaps?

"supported": [
      "linux/amd64",
      "linux/arm64",
      "linux/riscv64",
      "linux/ppc64le",
      "linux/s390x",
      "linux/386",
      "linux/mips64le",
      "linux/mips64",
      "linux/arm/v7",
      "linux/arm/v6"

Available to build with QEMU using docker container --platform=linux/arm64,linux/ppc64le.

Multiarch paths must not be hardcoded.

[ehfd]

Client should be split to library typescript component that does not use vuejs or any library and can be imported by any project. It should be as easy to integrate into custom project as embedding video player. Similar to how demodesk/neko-client is built, but without VueJs.

https://vitejs.dev/guide/#trying-vite-online might be a good complement to this concept. And I think React (for backward compatibility) or Svelte (for lightweightness and development speed) as the default interface has their points compared to Vue (the whole reason the project needs to be rewritten).

[m1k1o]

I already started and did the first step, where i upgraded demodesk/neko to vue3. Next step would be to remove vue3 as dependency for the core module and use vue only in the test client to speed up testing.

[ehfd]

We were using Vue 2 ourselves, so Vue 3 I guess, wouldn't hurt.

Reference performance:
● | TodoMVC-JavaScript-ES6-Webpack-Complex-DOM | 574.40 | ± | 214.15 (37.3%) | ms
● | TodoMVC-WebComponents | 152.79 | ± | 21.32 (14.0%) | ms
● | TodoMVC-React-Complex-DOM | 238.28 | ± | 19.36 (8.1%) | ms
● | TodoMVC-React-Redux | 248.94 | ± | 31.37 (12.6%) | ms
● | TodoMVC-Backbone | 185.90 | ± | 19.09 (10.3%) | ms
● | TodoMVC-Angular-Complex-DOM | 207.88 | ± | 11.95 (5.7%) | ms
● | TodoMVC-Vue | 174.76 | ± | 10.65 (6.1%) | ms
● | TodoMVC-jQuery | 1336.29 | ± | 122.84 (9.2%) | ms
● | TodoMVC-Preact-Complex-DOM | 89.70 | ± | 6.47 (7.2%) | ms
● | TodoMVC-Svelte-Complex-DOM | 85.70 | ± | 5.29 (6.2%) | ms
● | TodoMVC-Lit-Complex-DOM | 122.30 | ± | 9.74 (8.0%) | ms