Temporal Reprojection without Motion Vectors (FSR 2.0/DLSS 2.0)

[bloc97]

After contemplating for a long time about reprojection methods such as FSR 2.0 and DLSS 2.0 from the discussions #284 (comment) #363 (comment) #369 #151, etc., I found some relevant literature on temporal accumulation methods that do not require motion vectors.

Specifically the most relevant one was Hanika et al., Fast temporal reprojection without motion vectors, JCGT 2021, where the authors showed similar performance to analytical methods such as TAA.

Whether this would be a worthwhile direction for exploration should still to be discussed, but a game/engine-agnostic reprojection-based super-resolution algorithm would be extremely beneficial for gamers.

Three questions have to be addressed:

• Is it possible to jitter the camera/viewspace using sub-pixel mouse movements? Some engines might only move the camera when the mouse moves more than one pixel.
• How fast is this method compared to FSR on older GPUs? "Real-time" or the so-called "interactive" speed on newer and much more compute-focused GPUs might not be enough for older GPUs that are usually memory bandwidth limited in compute workloads.
• Whether the reprojection method focused on antialiasing would be applicable and extendable to a super-resolution problem. The authors gave some ideas about edge-aware upsampling, and I was thinking about sub-pixel alignment.

[MindSerene]

传闻中FSR2.0是基于多帧时间计算, 不知道会不会集成在A卡驱动中.以便于支持所有本机运行的应用.
Magpie本身是否可以通过获取并缓存过去1~5帧的画面来调用fsr2.0达到通用的效果

[Blinue]

FSR 2.0 需要历史帧以及运动矢量作为输入。前者可以通过缓存解决，bloc97 讨论的是后者

[Blinue]

Thank you for your investigation, this paper has benefited me a lot. However, the source code for FSR 2.0 has not yet been released, so the way it uses motion vectors is still unknown.

I'm basically just porting the algorithm rather than creating a new one, which I'm sure will be done by someone more specialized.

[bloc97]

Temporal reprojection shouldn't be too hard for me to try implementing, but I'm unsure the feasibility of out-of-engine jittering. Is it possible for magpie to move/rotate the camera slightly each frame? The movement has to be smaller than a pixel when projected on the screen.

Rotation of the camera is not as good as a translation of the camera by the engine itself but can be an acceptable approximation, as a sphere approximates its tangent line when the delta is small enough.

Also what are the performance implications of, let's say, caching 16 past frames and reading from those textures in a shader for older GPUs? If the performance would be very bad then this method would not work.

I'm less worried about the reprojection algorithm itself (there are plenty of literature on temporal super-resolution), but I'm doubting the feasibility of it without having access to the driver (in the case of Magpie).

[Blinue]

Is it possible for magpie to move/rotate the camera slightly each frame?

Magpie is just a simple simulation of the post-processing stage, it cannot affect the running state of the game, which is quite a dangerous behavior in online games.

Moving the camera from outside the game is difficult. Magpie does not know the game's mouse sensitivity configuration, nor does it know if the user has entered a menu, backpack, or similar page. In addition, many games read raw input directly from the mouse hardware, which makes it even more difficult to simulate mouse movement.

Also what are the performance implications of, let's say, caching 16 past frames and reading from those textures in a shader for older GPUs? If the performance would be very bad then this method would not work.

Starting with v0.9, Magpie requires DirectX feature level 11, which means we can assume that the user's graphics card is not from an ancient era. On these hardware, the biggest impact on speed is how many samples are performed, rather than how many textures are bound. Of course, caching more frames means more video memory is required, which is inevitable.

[bloc97]

I realize it is much harder than expected, but there might be solutions to these problems...

Magpie is just a simple simulation of the post-processing stage, it cannot affect the running state of the game, which is quite a dangerous behavior in online games.

It would be better for another standalone program to execute the mouse movements, this way magpie is completely safe to use while the other program would have a disclaimer associated with it and not to be used with anti-cheat software.

Moving the camera from outside the game is difficult. Magpie does not know the game's mouse sensitivity configuration, nor does it know if the user has entered a menu, backpack, or similar page. In addition, many games read raw input directly from the mouse hardware, which makes it even more difficult to simulate mouse movement.

Sensitivity could be estimated for free using the same method for estimating the motion vectors, and can be automatically adjusted. However that means either the user has to input the value manually given by the estimation from magpie, or there has to be a way for magpie to communicate with the program that moves the mouse.

For the problem of menus and static interfaces, I think we can just ignore it and use standard single-image super-resolution like FSR 1.0 to upscale any part of the image that does not move when the mouse moves. (This detection is also for free given we already estimated the motion vectors)

Starting with v0.9, Magpie requires DirectX feature level 11, which means we can assume that the user's graphics card is not from an ancient era. On these hardware, the biggest impact on speed is how many samples are performed, rather than how many textures are bound. Of course, caching more frames means more video memory is required, which is inevitable.

This is what I'm worried about the most because older GPUs might not have the required memory bandwidth to read something like 500 texture reads per pixel to do motion vector estimation. There's really not a lot of compute needed but memory bandwidth is very important...

[Blinue]

My current thinking is that it's not worth the effort for FSR 2.0. Still waiting for AMD to release the source code.

[MindSerene]

我目前的想法是，为FSR 2.0付出努力是值得的。仍在等待AMD发布源代码。

6月23日AMD开源了FSR2.0