Upscales and refines video (480x270 -> 1920x1080) using AI model.
Created for Joshimuz for upscaling GTA VCS streams.
Demo from a live stream (sadly, the plugin wasn't working properly).
The model is originally based on TecoGAN project by Mengyu Chu, et al.
This repository contains the following:
- avisynth plugin for video upscaling;
- OBS plugin for upscaling in real time;
- scripts and tools for model training.
Tested with Visual Studio 2022 (17.10.3) and g++ 12.3.0.
The project can be built using CMake. It is strongly recommended to use vcpkg for build-time dependencies.
Additionally you will need NVIDIA CUDA Toolkit and NVIDIA TensorRT SDK.
For building installer, you need to install Inno Setup.
Please put your models into data subfolder.
NVIDIA Maxine Broadcast SDK also uses TensorRT and it can lead to compatibility issues when OBS is trying to load it together with my plugin.
Therefore, it must be ensured that both use same TensorRT version.
To do this, you must use CUDA 11 and TensorRT 8.4 and set USE_NVVFX to ON.
All training scripts are located in here. The script supports training on CPU/GPU/TPU.
The model is trained using a two-step process:
- FRVSR training: generator and flow networks are trained together on MSE loss.
- GAN training: trains generator and flow networks with discriminator model using complex loss function.
The final model is exported as keras json config and weights in h5 format.
You can use either pairs of low resolution and high resolution images (recommended) or only high resolution images only (they will be downscaled for training). For training on TPU you should convert your dataset to tfrecords format.
Use export_model.py to convert your model to ONNX format. Then you can use scripts from here to optimize/tweak the models.
Check CONTRIBUTING.md. If you found any issues or need help related to this project, feel free to create issues on GitHub or contact my via email.