C++ WebRTC Bridge for Unreal Engine PixelStreaming
Unreal Engine released PixelStreaming with its 4.27 version. It enables remote visualization for small-frontend devices.
Synavis enables the coupling of simulation and ML tools to the Unreal Engine by leveraging PixelStreaming as data source.
See our video:
I am currently migrating the Synavis Plugin to accomodate dynamic workflows. Read more about it in the news post and in the living doc.
- DataConnector and MediaReceiver: Connect DL Frameworks and Simulations to UE and interact with the scene
- Synavis: Setup connections via a bridged network (like a TURN server) via port relays. This is intended for HPC systems but might be used together with Putty to bridge to client PCs
- Syanvis Signalling Server: Load Handling and Connection Management
- PySynavis: Python Coupling of all revelant functionality
- Automatic Connection setup and infrastructure-focussed handling is in works
- Decoding is in works, with UE allowing the parsing of software-based encoders, we aim to support these primarily, as they are not dependent on a specific GPU.
- Our course "Virtual Worlds for Machine Learning" provided some in-depth information on the framework and is the newest tutorial. Contact us for more information.
- The wiki contains all current information and is being edited more frequently than this repository: Synavis Wiki
I would greatly appreciate help with this project. It is an integral part of my thesis, but not the main focus of it. I am working in Berlin time and will respond during work hours when I can spare time.
I would like to acknowledge funding provided by the German government to the Gauss Centre for Supercomputing via the InHPC-DE project (01—H17001).
Dirk Norbert Helmrich, Felix Maximilian Bauer, Mona Giraud, Andrea Schnepf, Jens Henrik Göbbert, Hanno Scharr, Ebba Þora Hvannberg, Morris Riedel, A Scalable Pipeline to Create Synthetic Datasets from Functional-Structural Plant Models for Deep Learning, in silico Plants, 2023;, diad022, https://doi.org/10.1093/insilicoplants/diad022
Baker et al. "Adapting Agricultural Virtual Environments in Game Engines to Improve HPC Accessibility" 6th Nordic e-Infrastructure Collaboration Conference, NeIC 2024, Tallinn, Estonia, May 27–29, 2024, Communications in Computer and Information Science, vol 2398. Springer: 10.1007/978-3-031-86240-3_11
Dirk N Baker, Mona Giraud, Jens Henrik Göbbert, Hanno Scharr, Morris Riedel, Ebba Thóra Hvannberg, Andrea Schnepf, Virtual World Coupling with Photosynthesis Evaluation for Synthetic Data Production, in silico Plants, 2025;, diaf018, https://doi.org/10.1093/insilicoplants/diaf018