Scrcpy Standalone Server Latency question

[Fulcinator]

I started trying to use Scrcpy recently to capture my android screen to mirror it to my computer. I went very smoothly and it is actually better than any other solution I found so far.

However, my final goal would be to use the video stream not inside a standalone window, but incorporated into a "virtual environment" using Unreal Engine 5. Recording the window itself and streaming the media source to UE5 indeed would be very easy, but increase the streaming latency.

I started looking into Standalone server to check if I could simply create a video stream that can be incorporated into UE5, which requires a URL when capturing it. I managed to use the standalone server solution (following the doc/develop.md instructions), but the latency experieced is very high, completely different to the one experienced when using scrcpy itself.
Do you think this is due to container encapsulation (such as MP4)? Have you got any suggestion (even customizing part of the codebase) on how I can encapsulate the video stream directly in UE5, or eventually efficienlty stream it with the lowest latency possible?

Thank you very much for your work!

[rom1v]

You need a custom player that receives input packets, decodes them and displays the resulting frames as fast as possible, without buffering. An existing video player will typically never do that.

And you must also take care to avoid an additional 1-frame latency when receiving packets: #646.

[Fulcinator]

Thank you for the quick reply. Do you think it is feasible to integrate the client codebase of this project inside UE5?

[rom1v]

🤷