Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SFU Server: Multiple users connected simultaneously, implement view-only password, isolated sessions #39

Open
ehfd opened this issue Jul 18, 2022 · 13 comments
Labels
enhancement New feature or request help wanted External contribution is required transport Underlying media or data transport protocols web Web components including gst-web

Comments

@ehfd
Copy link
Member

ehfd commented Jul 18, 2022

Apache Guacamole and noVNC provide access to multiple users simultaneously to one screen. They also have functionalities which allow users to set view-only passwords for view-only screen sharing.
It is required to investigate the feasibility of this function for the Selkies project to be used more widely as a stable remote desktop solution.

Also: what about isolated sessions for multiple users, where throwaway desktops are created and scrapped for different users?
Such environment probably requires a lot more things than just Selkies-GStreamer, but the scope for the remote desktop system is addressed here.

@ehfd
Copy link
Member Author

ehfd commented Aug 25, 2022

Relevant code: https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/signalling_web.py
Should be addressed with #27.
The signalling server must be rewritten to accommodate multiple peers. Anyone with experience in WebRTC development can do it.

@ehfd ehfd changed the title Multiple users at a time and view-only passwords Multiple users at a time Aug 25, 2022
@ehfd ehfd changed the title Multiple users at a time Multiple users connected at a time Sep 7, 2022
@ehfd
Copy link
Member Author

ehfd commented Sep 7, 2022

This might be addressed with #48.

@nums
Copy link

nums commented Sep 7, 2022

To be able to send stream to others participants, I think the ideal way is to send stream to a WebRTC SFU server.
Neko project is using Pion WebRTC (https://github.com/m1k1o/neko) to achieve that.
Webrtcbin must be able to interface with SFUs (there is a WHIP client that uses webrtcbin https://github.com/meetecho/simple-whip-client)
Once done, the main problem is the management of the quality of the streaming with the recovery of Nack events, PLI etc and this can produce lags or a noticeable drop in the quality of the video.

The mediasoup documentation explains this very well:
https://mediasoup.org/documentation/v3/scalability/#one-to-many-broadcasting

@ehfd
Copy link
Member Author

ehfd commented Sep 7, 2022

@nums Thank you for the information! We are currently focusing on improving the documentation for more contributor accessibility. We have a goal for this project to potentially attract community contributors from every relevant field, from experienced GStreamer/WebRTC/conferencing/media/graphics software developers to video power users who simply know a lot of about optimal encoder parameters, and Python/JS learners.

@ehfd
Copy link
Member Author

ehfd commented Sep 9, 2022

https://github.com/centricular/webrtcsink instead of webrtcbin may solve this issue as this has its own signalling server.

@nums
Copy link

nums commented Sep 9, 2022

It can, but they incidate that :

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

So it is possible, but there will be a big use of resources. Some certain nvidia card have a limitation of number of encodings so it can be a problem.

I'm working on my side to use webrtcbin with a SFU with this "bridge" https://github.com/meetecho/simple-whip-client and to stream the desktop screen to an SFU like mediasoup. I just have a little issue on the compilation with the selkies gstreamer version, but "work in progress"

@ehfd
Copy link
Member Author

ehfd commented Sep 9, 2022

@nums I guess that you are working on a contribution. We greatly thank your time.

@nums
Copy link

nums commented Sep 9, 2022

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story

@ehfd
Copy link
Member Author

ehfd commented Sep 9, 2022

Yes I'm trying to share the screen in "readonly" mode, but to be able to take the control il will be an other story.

That's already something we really need. Thanks a lot!

@ehfd ehfd added documentation Documentation improvements or additions help wanted External contribution is required enhancement New feature or request transport Underlying media or data transport protocols and removed documentation Documentation improvements or additions labels Oct 12, 2022
@ehfd ehfd changed the title Multiple users connected at a time Multiple users connected simultaneously Oct 13, 2022
@ehfd ehfd changed the title Multiple users connected simultaneously Multiple users connected simultaneously, implement view-only password Jan 18, 2023
@ehfd
Copy link
Member Author

ehfd commented Aug 22, 2023

Relevant: #48 (comment)

@ehfd ehfd changed the title Multiple users connected simultaneously, implement view-only password Multiple users connected simultaneously, implement view-only password, isolated sessions Oct 12, 2023
@ehfd ehfd added the web Web components including gst-web label Oct 12, 2023
@ehfd
Copy link
Member Author

ehfd commented Apr 3, 2024

WebRTCSink is a wrapper for WebRTCBin, where settings are available in https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/blob/main/net/webrtc/src/webrtcsink/imp.rs

The issue of WHIP/WHEP is that there is no DataChannel for keys, cursors, and other information. Only read-only access would be possible without other transport protocols.

Thus, the immediate property might rather be a multi-user SFU server.

Need contribution for the SFU server, perhaps utilizing GStreamer's own signalling server.

https://gstreamer.freedesktop.org/documentation/rswebrtc/index.html#usage

@ehfd ehfd changed the title Multiple users connected simultaneously, implement view-only password, isolated sessions SFU server: Multiple users connected simultaneously, implement view-only password, isolated sessions Apr 15, 2024
@ehfd ehfd changed the title SFU server: Multiple users connected simultaneously, implement view-only password, isolated sessions SFU Server: Multiple users connected simultaneously, implement view-only password, isolated sessions Apr 15, 2024
@PMohanJ
Copy link
Member

PMohanJ commented Jul 3, 2024

The current architecture of selkies-gstreamer consists of two webrtc connections(video and audio) for one single session to a user. So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

If so, considering the below note, wouldn't it be resource intense even for as little as 2-3 users?

It is important to note that at this moment, encoding is not shared between consumers. While this is not on the roadmap at the moment, nothing in the design prevents implementing this optimization.

@ehfd
Copy link
Member Author

ehfd commented Jul 3, 2024

So using plugin webrtcsink would result in 2*n connections (of course, all are handled by webrtcsink itself) for n number of session?

This is called an MCU server.

An SFU server distributes the same stream(s) to many clients. I still think webrtcsink's signaling server is still capable of doing this.

https://forasoft.medium.com/p2p-vs-mcu-vs-sfu-what-to-choose-for-a-video-conference-app-fa5cb4a5dd6e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted External contribution is required transport Underlying media or data transport protocols web Web components including gst-web
Projects
None yet
Development

No branches or pull requests

3 participants