DLSS in HDRP - Deep Learning Super Sampling integration #3484

kecho · 2021-02-12T15:47:38Z

Integrating NVIDIA's Deep Learning Super Sampling into HDRP

Purpose of this PR

Adding new DLSS upscaler into HDRP.
Adding the concept of upscaling before post process.
Post process pipeline massaging and refactor of destination size. Added the concept of post process size scales, for sampling and rendering, since now the dynamic resolution upscaler could run before or after post processes.
Adding new UI controls of DLSS
Adding new tests images for DLSS
Added new debug panel in the render pipeline analysis pane, to check the internal state of NVIDIA unity plugin.

Testing status:

To test, just get the latest trunk version, which contains DLSS backend features.

The following steps have been tested locally:

Open the package manager, select in the filter "Built-in" , ensure to enable the "NVIDIA" core native module in your project.
Ensure you dont have render doc on otherwise, the feature wont be detected. In the HDRP quality settings, right under Dynamic Resolution section, enable "tick on the Enable NVIDIA Deep Learning Super Sampling Checkbox"
Ensure your desired camera has Dynamic Resolution Enabled (check in the inspector the checkbox is on)
Read the info box that will appear. This info box will let you know if DLSS is detected or not on this session. If not, then the fallback regular antialiasing is used, and the standard DRS path is executed.
Inspect / play around with software/hardware dx12/dx11/vulkan APIs of the effect

This effect must not be present in any other platform other than Windows, and in the case any GPU other than NVIDIA's is present, the effect must not be enabled.

If you are measuring performance, please ensure that you are looking at GPU time, using the GPU profiler, and not the CPU profiler. Note too that DLSS wont make your frame run faster unless your scene is highly bound by gpu draw calls.
One example is spaceship demo.

Known Testing Issues:

We have 1 known crash, vulkan on intel CPU is now crashing %100 of the time, this is being addressed in a separate PR:
https://ono.unity3d.com/unity/unity/pull-request/125566/_/graphics/fix-nvapi-vulkan-crash

com.unity.render-pipelines.core/Runtime/Textures/RTHandleSystem.cs

com.unity.render-pipelines.high-definition/Runtime/ShaderLibrary/ShaderVariables.hlsl

com.unity.render-pipelines.core/Runtime/Textures/RTHandleSystem.cs

...ity.render-pipelines.high-definition/Runtime/PostProcessing/PostProcessSystem.RenderGraph.cs

com.unity.render-pipelines.high-definition/Runtime/PostProcessing/PostProcessSystem.cs

com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/Camera/HDCamera.cs

FrancescoC-unity · 2021-02-15T09:40:14Z

...nity.render-pipelines.high-definition/Runtime/RenderPipeline/HDRenderPipeline.RenderGraph.cs

+                passData.shaderVariablesGlobal = m_ShaderVariablesGlobalCB;
+                passData.postProcessWidth  = postProcessWidth;
+                passData.postProcessHeight = postProcessHeight;
+                builder.AllowPassCulling(false);


No need for this line

Done. May I ask then, when are render passes culled? Since this pass is not using any render targets (are render passes withouth any targets automatically not culled?)

This indeed feels wrong on second read (was ealry morning first time around :D ) Though I see most of the passes we do that just push constants don't have that set.

Paging in rendergraph owner @JulienIgnace-Unity

@JulienIgnace-Unity I still dont know, but I do think I should leave the AllowPassCulling here right?

Wrong @JulienIgnace-Unity :D

@kecho I think you referenced the wrong Julien... 😃

com.unity.render-pipelines.high-definition/Runtime/ShaderLibrary/ShaderVariables.hlsl

com.unity.render-pipelines.core/Runtime/Textures/RTHandleSystem.cs

com.unity.render-pipelines.core/Runtime/Common/DynamicResolutionHandler.cs

JordanL8 · 2021-04-29T10:47:43Z

com.unity.render-pipelines.high-definition/Documentation~/Default-Settings-Window.md

@@ -47,5 +54,6 @@ Use this section to select which custom [Diffusion Profiles](Diffusion-Profile.m
 | --------------------------| ------------------------------------------------------------ |
 | Shader Variant Log Level              | Use the drop-down to select what information HDRP logs about Shader variants when you build your Unity Project. • Disabled: HDRP doesn’t log any Shader variant information.• Only HDRP Shaders: Only log Shader variant information for HDRP Shaders.• All Shaders: Log Shader variant information for every Shader type. |
 | Lens Attenuation Mode                 | Set the attenuation mode of the lens that is used to compute exposure. With imperfect lens some energy is lost when converting from EV100 to the exposure multiplier, while a perfect lens has no attenuation and no energy is lost. |
-| Light Layer Names                     | The name displayed on Lights and Meshes when using [Light Layers](Light-Layers.md). |
-| Decal Layer Names                     | The name displayed on decals and Meshes when using [Decal Layers](Decal.md). |
+| Use DLSS Custom Project Id            | Controls whether to use a custom project ID for the NVIDIA Deep Learning Super Sampling module. If you enable this property, you can use **DLSS Custom Project Id** to specify a custom project ID.<br/>This property only appears if you enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project. |


From reading the stuff from Paul in slack, I think the first bit is fine. We should just add information about what happens when disabling the property. So:
Controls whether to use a custom project ID for the NVIDIA Deep Learning Super Sampling (DLSS) module. If you enable this property, you can use **DLSS Custom Project Id** to specify a custom project ID. If you disable this property, Unity generates a unique project ID. <br/>This property only appears if you enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project.

I also think it's important to add info about the ID being unique and why. I'm not sure if it should or must be unique though. I've asked in slack

Initial changes for dynamic resolution rendering pre post processes. Integrating bug fixes and fixing post process logic. Adding abstraction for resolution scheduling of camera. Fixing some file formatting. Setting camera schedules during certain parts of the frame Renamed the whole thing to resolution group. Much prettier. Some new line Setting camera resolution passes and updating constant buffer between passes. Adding dynamic resolution swaps spots, to regress previous behaviour. Adding the resolution group setting during scene upsampling. Undoing the constant buffer approach. Applying a separate scale to post process inputs (color, velocity and depth). Only on UberPost shader. Fixing some issues with the resolution size calculation, removing prescaling. Applying scale for post process in a more elegant manner. Additional style fixes, fixing private / public things and other review feedback UI Changes to easily enable / disable pre post upscaler. More ui prettification, adding possibility of detecting pass Reworking UI based on feedback. Adding feature detection mechanism. Formatter More clean comment into scaling function. Moved post process RT scale to camera instead of drs handler. Renamed resolution groups to something more appropiate. Renaming viewport to viewportSize. Getting Physically based dof to work (without having TAA or jittering working yet). Fixing rebase problems. Fixing CAS changes lost from rebase. Support for depth of field Prepost upsampling. Fixing some issues with jittering on coc reprojection. Fixing bloom prefiltering for drs prepost Support for motion blur drs prepost Panini projection and fixing CAS again. Updating fxaa scales for consistency. Remove round, since this might affect results if the targets are flipped. Reapplying changes from merge First commit of DLSS: * Creating DLSS rendering module * Using plugin and bringing NVDIA's plugin wrapper * Adding Post Process component For now its hacked in and only works on 1 to 1 resolution Adding missing files. Ensuring DLSS is initialized properly, and works only on the GameView camera. Connecting dlss to HDRP, now working on many resolutions. To enable it, DLSS must be set as the upsampling filter. Connecting sharpness feature to dlss. Fixing resolution issues on motion vectors. Adding depth buffer in intermediate prepare pass. New plugin. DLSS Removing Prepare input pass. No longer needed. Disabling the dynamic resolution boolean for now, since its not working properly. Fixing sharpness caching. Compilation issue Moving DLSS to be applied as an AA solution instead. Supporting more than 1 camera for render scripting pipeline. Handling destruction / reconstruction of rendering pipeline and addressing any possible memory leaks. New naming convension and refactor of Nvidia API DX12 Support plugin Line endings Support for DLSS without plugin, dlls coming from player. Ensuring that HDRP can compile without DLSS. Initial UI changes for DLSS, applying DLSS before post effects. Updating plugin detection, and basic ui work to display plugin state. Adding missing check Formatting UI messaging feedback. Taking out dlss as a filter from the list. Better name for ui switch to fallback. Removing useless files. Fixing some issues with rebase. Exposing quality settings in the UI. Exposing DLSS settings for sharpness and quality. Left over code Adding debug panel for nvidia module Stale results in debug window, and formatting Formatting Nullcheck upon destruction of dlss feature. Showing versions of plugins Fixing memory leaks by resetting the device. This can happen if settings are tweaked New api changes, safer texture uploading, more costly but thread safe. Adding support for multipass XR Fixing stale parameters, and fixing camera reset. Support for XR singple pass and multipass modes. Formatting Adding tests for DLSS and fixing issues with a warning on unset color texture Adding post processes / bloom / sky to test scene. Since DLSS must run always, adding API so we dont disable DRS when the requested frame % is 100. formatting Updating namespace of nvidia DLSS package Adding API to toggle / control dlss programmatically Update of API, so we can use pre exposure and bias textures. Added also API mechanism to detect compatibility Bad inverted texture coords on final output Changelog updates Refactoring application of scale so that we dont create inconsistent results when we dont use DRS Adding safety check for nvidia vendor, also fixing potential leaks from WeakReference living too high in the key Addressing memory leaks (accessing objects from functors cause leaks) Allowing a few frames before taking a test snapshot. Description for render frame variable Removing render flag setting, since DLSS must be twinkered with before any cameras can start, due to how DRS is updated Feedbck from francesco, making conditionals more explicit for settings of the DRS Adding new API for optimal settings User interface work and new API for DynamicResolutionHandler to set system scaler UX work for DLSS, cleaning settings and renaming parameters more consistently Removing wrong call to hd Camera SetPostProcessScreenSize, this could cause the camera to render the wrong resolution Fixing invalid parameter for physical burte for dof UI feedback, correct indentation and parameter renaming to be more consistent. Removing files not needed accidentally checked in Add percentages text. Fixing dumb performance issues with wrong call to UI Test images Updated new API Formatting of refactor New api for plugin control Renaming of some structures in dlss api merge conflict API feedback update Documentation & public API mangling Changing depth filter to Nearest Feedback rework, moving nv debug view, renaming and docs Reviewed HDRP Asset and Camera documentation Documentation for the HDRP pipeline debug window Reviewed DLSS debug view documentation Exposing custom project ID for nvidia DLSS DLSS feedback from seb: better eplanation of before/after post, removing changes to lock file and nicer default id for dlss project id Updated documentation suggestion by Lewis Updating cam Updating defines and update of NVUnityPlugin Documentation update Merge camera UI refactor

…rp settings

remi-chapelain · 2021-05-06T14:57:56Z

com.unity.render-pipelines.high-definition/Documentation~/HDRP-Asset.md

@@ -79,6 +79,10 @@ These settings control the draw distance and resolution of the decals atlas that
 | **Property**                    | **Description**                                              |
 | ------------------------------- | ------------------------------------------------------------ |
 | **Enable**                      | Enable the checkbox to make HDRP support dynamic resolution in your Unity Project. |
+| **- Enable DLSS**               | Enable the checkbox to make HDRP support NVIDIA Deep Learning Super Sampling (DLSS).<br/>This property only appears if you enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project. |
+| **-- Performance Quality**      | Use the drop-down to select which performance mode DLSS operates on. The options are:<br/>&#8226; **Balanced**: - Balances performance with quality.<br/>&#8226; **MaxPerf**: - Fast performance, lower quality.<br/>&#8226; **MaxQuality**: - High quality, lower performance.<br/>&#8226; **UltraPerformance**: - Fastest performance, lowest quality.<br/>&#8226; **UltraQuality**: - Highest quality, lowest performance. |


| -- Performance Quality |

I Believe this has change to "mode" now, to better match what we're doing in raytracing perf/quality mode

Also, I believe there's only 4 mode

Max Quality

Balanced

Max Perf

Ultra Perf
Ultra quality doesn't exists

…tance of DLSS

…es/Graphics into HDRP/DRSPrepost

remi-chapelain · 2021-05-06T14:59:13Z

com.unity.render-pipelines.high-definition/Documentation~/HDRP-Asset.md

@@ -79,6 +79,10 @@ These settings control the draw distance and resolution of the decals atlas that
 | **Property**                    | **Description**                                              |
 | ------------------------------- | ------------------------------------------------------------ |
 | **Enable**                      | Enable the checkbox to make HDRP support dynamic resolution in your Unity Project. |
+| **- Enable DLSS**               | Enable the checkbox to make HDRP support NVIDIA Deep Learning Super Sampling (DLSS).<br/>This property only appears if you enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project. |
+| **-- Performance Quality**      | Use the drop-down to select which performance mode DLSS operates on. The options are:<br/>&#8226; **Balanced**: - Balances performance with quality.<br/>&#8226; **MaxPerf**: - Fast performance, lower quality.<br/>&#8226; **MaxQuality**: - High quality, lower performance.<br/>&#8226; **UltraPerformance**: - Fastest performance, lowest quality.<br/>&#8226; **UltraQuality**: - Highest quality, lowest performance. |


Also, I believe there's only 4 mode

Max Quality

Balanced

Max Perf

Ultra Perf
Ultra quality doesn't exists

remi-chapelain · 2021-05-06T15:01:46Z

com.unity.render-pipelines.high-definition/Documentation~/HDRP-Asset.md

+| **- Enable DLSS**               | Enable the checkbox to make HDRP support NVIDIA Deep Learning Super Sampling (DLSS).<br/>This property only appears if you enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project. |
+| **-- Performance Quality**      | Use the drop-down to select which performance mode DLSS operates on. The options are:<br/>&#8226; **Balanced**: - Balances performance with quality.<br/>&#8226; **MaxPerf**: - Fast performance, lower quality.<br/>&#8226; **MaxQuality**: - High quality, lower performance.<br/>&#8226; **UltraPerformance**: - Fastest performance, lowest quality.<br/>&#8226; **UltraQuality**: - Highest quality, lowest performance. |
+| **-- Use Optimal Settings**     | Enable the checkbox to make DLSS control the Sharpness and Screen Percentage automatically. |
+| **-- Sharpness**                | Use the slider to set the pixel sharpness of the DLSS upscaler. |


I suggested another tooltip on editor side to prevent us using the same word (shaprness) we're trying to define and give a bit more info on what's at stake here.

"Controls how the DLSS upsampler will render edges on the image. More sharpness usually means more contrast and clearer image but can increase flickering and fireflies. This setting is ignored if ‘Use Optimal Settings’ is checked."

remi-chapelain · 2021-05-06T15:02:28Z

com.unity.render-pipelines.high-definition/Documentation~/HDRP-Camera.md

@@ -37,6 +37,12 @@ The HDRP Camera shares many properties with the Unity's [standard Camera](https:
 | **Allow Dynamic Resolution**    | Enable the checkbox to make this Camera support dynamic resolution for buffers linked to it. |
 | **Fullscreen Passthrough**      | Enable the checkbox to make this Camera skip rendering settings and directly render in full screen. This is useful for video. |
 | **Custom Frame Settings**       | Enable the checkbox to override the default [Frame Settings](Frame-Settings.md) for this Camera. This exposes a new set of Frame Settings that you can use to change how this Camera renders the Scene. |
+| **Allow DLSS**                  | Controls whether this camera can use NVIDIA Deep Learning Super Sampling (DLSS). To use DLSS, enable the NVIDIA package (com.unity.modules.nvidia) in your Unity project and enable DLSS in your [HDRP Asset](HDRP-Asset.md). |
+| **- Use Custom Quality**        | Indicates whether this Camera overrides the DLSS quality mode specified in the [HDRP Asset](HDRP-Asset.md). If you enable this property, HDRP uses the below **Performance Quality** property. |
+| **-- Performance Quality**      | Specifies the performance quality mode override for performance quality mode for DLSS.The options are:<br/>&#8226; **Balanced**: - Balances performance with quality.<br/>&#8226; **MaxPerf**: - Fast performance, lower quality.<br/>&#8226; **MaxQuality**: - High quality, lower performance.<br/>&#8226; **UltraPerformance**: - Fastest performance, lowest quality.<br/>&#8226; **UltraQuality**: - Highest quality, lowest performance.<br/>This property only appears if you enable **Use Custom Quality**. |


Same comment here about naming of the property and only 4 mode instead of 5

remi-chapelain · 2021-05-06T15:05:28Z

...nity.render-pipelines.high-definition/Documentation~/deep-learning-super-sampling-in-hdrp.md

+To change the DLSS quality mode for your whole project:
+
+1. Select the HDRP Asset that has DLSS enabled and view it in the Inspector.
+2. Go to **Rendering** > **Dynamic Resolution** > **DLSS** and set the **Performance Quality** property to the quality mode you want.


Same comment about property naming "mode"

remi-chapelain · 2021-05-06T15:05:37Z

...nity.render-pipelines.high-definition/Documentation~/deep-learning-super-sampling-in-hdrp.md

+
+1. In the Hierarchy or Scene view, select a Camera and view it in the Inspector.
+2. Select **Use Custom Quality**.
+3. Set the **Performance Quality** property to the quality mode you want.


Same comment about property naming "mode"

remi-chapelain

PR is in good shape. Go for merge once little things from documentation will be updated. 💚

Did a few performances tests to ensure that there's a net gain (up to 35/40%) even though we trust Nvidia for this.

A few things were improved including :

Issue about quality settings not taken into account
More info on tooltips
Quality settings naming for consistency with RTX naming
Ease of use for installing Nvidia package (without having to go to package manager) and switching architecture to 64 bits

Will enter issues for those 2 concerns that doesn't block nor prevent anyone from using DLSS.

The order of the quality settings is misleading. It would make more sense to go from Best perf to best quality (or inverse) instead of jumping from perf to balance, to quality, to perf again. This require a cpp PR, can be done later, not blocking anything.
Also found an issue about black pixels on one frame when switching DLSS ON that would be related to how we handle RT in Vulkan.

Here's the test document for more information about what's tested and discussions.

…a quality

…conflicting test ids

kecho requested review from sebastienlagarde, JulienIgnace-Unity and FrancescoC-unity February 12, 2021 15:47

github-actions bot added HDRP SRP labels Feb 12, 2021

kecho force-pushed the HDRP/DRSPrepost branch from 321e96b to 7dd7032 Compare February 12, 2021 20:33

kecho force-pushed the HDRP/fixing-CAS-Resolution-Issues branch from bc93b02 to 77f3a74 Compare February 12, 2021 20:34