Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream encoding mode for desktop and monitor modes #3872

Closed
totaam opened this issue May 28, 2023 · 3 comments
Closed

stream encoding mode for desktop and monitor modes #3872

totaam opened this issue May 28, 2023 · 3 comments
Labels
encoding enhancement New feature or request

Comments

@totaam
Copy link
Collaborator

totaam commented May 28, 2023

Two different ways of implementing this:

  • re-using the GStreamer CaptureAndEncode (gstreamer shadow capture #3750) to bypass most of the xpra rate, speed and quality controls and simply push video frames as packets:
    class CaptureAndEncode(Capture):
    """
    Uses a GStreamer pipeline to capture the screen
    and encode it to a video stream
    """
  • re-using our own WindowVideoSource
    This one has the benefit of already handling all the bandwidth controls, including bandwidth limit, automatic downscaling, etc.
@totaam totaam added enhancement New feature or request encoding labels May 28, 2023
totaam added a commit that referenced this issue May 30, 2023
@totaam
Copy link
Collaborator Author

totaam commented May 30, 2023

The commit above works reasonably well in most cases, but:

  • with the gstreamer encoders, we end up making an extra memory copy of the xshm data: gstreamer zero copy #3754
  • we could do something smarter when the number and size of the damage regions updated is small: just paste them over the previous image wrapper of the window - is it worth the effort and complications?

Using the CaptureAndEncode approach would save one memory copy.

@totaam
Copy link
Collaborator Author

totaam commented Jun 9, 2023

With some preparatory work already done in 9a1ed42, 057e240 adds a pure GStreamer encoder that injects packets directly using the same code as the gstreamer shadow capture (#3750)

This mode is hidden for now because it isn't ready:

  • the client must use --encoding=stream - just like above
  • the server must be started with XPRA_STREAM_MODE=gstreamer

Many more changes are needed for this new mode:

  • handle the window being resized - not working yet - not sure why, too many events? should be throttled anyway
  • figure out why the damage extension is not working for the ximagesrc gstreamer element - perhaps add a gate that we control so we can decide when to start processing the next frame? similar to a leaky queue? ideally, blocking ximagesrc from producing more frames if we're not going to use them
  • better sanity checks, the window may be resized and could become smaller than what the encoder can deal with - though monitor and desktop mode should already have a sensible minimum size
  • if this is enabled for seamless mode, it should only be used for video windows, or at least not for text ones, not for popups, etc..
  • make it easier to customise the pipeline used so we can use GPU encoders
  • make it easier to enable this mode?

@totaam totaam changed the title "dumb" streaming mode for desktop and monitor modes stream encoding mode for desktop and monitor modes Jul 18, 2023
@totaam
Copy link
Collaborator Author

totaam commented Apr 19, 2024

Now automatically enabling encoding=stream: 509af1a
And also automatically using the gstreamer encode if this gives us GPU acceleration.

@totaam totaam closed this as completed Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
encoding enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant