Still Failed to create Vulkan device, vr -3 on Ampere

[RoyShapiro]

Hello again.
This is in continuation of #488.
Nvidia driver 455.50.05 was supposed to fix the err:vkd3d_create_vk_device: Failed to create Vulkan device, vr -3 error on Ampere GPUs, namely RTX 3090.
However, with that driver on a freshly booted \ rebooted system, the problem persists in all D3D12 apps and games that use vkd3d-proton, this time regardless of whether they are run with or without Steam Runtime, that caused the problem previously.

Wine \ proton version, settings, as well as vkd3d-proton build, if it is a post-b4bc927 build, do not seem to fix the issue.

However, for some extremely weird and peculiar reason, if a vkd3d-proton test suite is natively built by doing the following:

meson -Denable_tests=true build
cd build
ninja

and then run at any time after system boot using:
./tests/d3d12

The problem goes away, and stays gone (everything works) until the next reboot, whereby it reappears and stays until ./tests/d3d12 (output attached output.txt) is run again.

It would seem as if running d3d12 native test somehow affects certain values in memory, that cause the issue to go away.
This is extremely weird and unexpected behavior. Not sure if it is a driver issue or vkd3d-proton's, since running that test works as a reliable workaround for an unknown reason.

Same system, software and hardware config as in #488, except the driver which is 455.50.05.

Originally posted by @RoyShapiro in #488 (comment)

@liam-middlebrook Liam, tagging you, since this may have to do with the driver, pardon for any inconvenience.

P.S. Sorry for opening a duplicate issue, the previous one got Closed before I could fully test the proposed solution and I couldn't reopen it. I'm relatively new to GitHub and don't know how visible the comments on the Closed issues are.

[HansKristian-Work]

I think for the next release we'll just need to temporarily disable RT extensions altogether.

[RoyShapiro]

While this will help those currently unable to play the games, wouldn't it reduce the amount of test cases in the wild? Given that there were few reports on Ampere cards and this particular oddness before, I fear more strangeness may emerge. On the side note, it's very interesting what could be causing this behavior under the hood.

…

On Fri, Feb 12, 2021 at 1:22 PM Hans-Kristian Arntzen < ***@***.***> wrote: I think for the next release we'll just need to temporarily disable RT extensions altogether. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#500 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQX4ZGSF6OJUR6DOLNJ3NMDS6T6NJANCNFSM4XKGYSPA> .

[liam-middlebrook]

Just to confirm you’re only seeing this vkCreateDevice() failure on first launch with Proton/WINE, right?

Could you grab NVIDIA bug report logs for the following states?

1. After a fresh reboot without playing any games
2. After launching a game which uses vkd3d-proton (the version that has Ray Tracing extensions enabled)
3. After launching a native application which uses the Vulkan Ray Tracing extensions (either vkd3d-proton tests or Quake II RTX should do)
4. Repeat step 2

Either attach the logs here or in an email to linux-bugs [at] nvidia.com

[RoyShapiro]

Hi, Liam! I've sent you logs to linux-bugs [at] nvidia.com as requested. The message should have the topic "Liam Vkd3d-Proton Issue 500 logs as requested (RoyShapiro)" . Can confirm that running q2rtx has the same effect (makes everything work) as running d3d12 native vkd3d-proton test. Strange that other native Vulkan apps don’t have the same effect (but I guess it has to do with Raytracing being enabled). All files have been renamed for your convenience — filenames contain when the log had been taken such as nvidia-bug-report.log.vkd3d-protongame-success-afterq2rtx.gz. Also included are Lutris logs for GOG version of Cyberpunk 2077 (with which tests were performed): before running q2rtx on fresh boot (fail) and after running q2rtx (success). Let me know if you need anything else!

…

On Sat, Feb 13, 2021 at 10:19 PM Liam Middlebrook ***@***.***> wrote: Just to confirm you’re only seeing this vkCreateDevice() failure on first launch with Proton/WINE, right? Could you grab NVIDIA bug report logs for the following states? 1. After a fresh reboot without playing any games 2. After launching a game which uses vkd3d-proton (the version that has Ray Tracing extensions enabled) 3. After launching a native application which uses the Vulkan Ray Tracing extensions (either vkd3d-proton tests or Quake II RTX should do) 4. Repeat step 2 Either attach the logs here or in an email to linux-bugs [at] nvidia.com — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#500 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQX4ZGRXQ2EI5ZZJSVOKMULS63GDPANCNFSM4XKGYSPA> .

[liam-middlebrook]

Thanks Roy. From the logs you attached it looks like the nvidia_uvm.ko kernel module is not being loaded properly. This kernel module is a requirement for a few different driver features. I was able to also confirm this behavior when running Quake II RTX using the Steam Linux Runtime (SLR) compatibility tool, in addition to Proton 5.13, and Proton Experimental.

I believe changes will need to be made to the SLR in order to include /usr/bin/nvidia-modprobe which is a helper binary included with the driver to allow as-needed loading of some of our kernel modules. In the meantime, I suggest running sudo modprobe nvidia_uvm before trying to launch any Vulkan application which will make use of the NVIDIA or Khronos Ray Tracing extensions. From the logs you sent me it appears that you may have nvidia.ko, nvidia_modeset.ko, and nvidia-drm.ko included in your initramfs, so you may want to also include nvidia_uvm.ko there in order to avoid having to manually load it, at least until SLR is able to complete the Vulkan driver's call to nvidia-modprobe.

[RoyShapiro]

Thanks, Liam. I haven't inspected the contents of or made any manual changes to initramfs on this system yet, as until now there was no reason to do so. It will be interesting to look at it's current contents. That said, considering I frequently update my drivers and have a boot-time script for VMWare set up anyway, it would be convenient enough for me to just include the modprobe nvidia_uvm command there for the time being. Got a question, though. If nvidia-modprobe is to be called from SLR, what would that mean for Vulkan Raytracing apps being launched from vanilla Wine that, AFAIK, doesn't use SLR? Will they still fail, unless user loads nvidia-modprobe / nvidia_uvm manually? As an example, a user might be trying to launch a game, such as Deliver Us The Moon, that does make use of Raytracing, and is available from GOG, hence there's a possibility that user doesn't have Steam installed, however small that is, using some future version of vkd3d-proton that will support DXR. The obvious workaround for now is to just launch it in D3D11 mode, since, in that game, it is available. But future games might drop it. Just wondering aloud.

[RoyShapiro]

With drivers version 470.57.02 the current correct (and simplest) solution is to run
nvidia-modprobe -u -c=0 (add sudo as needed)
at DE launch time (can be automated by putting it in a script set to autorun), as mentioned here:
#711 (comment)
by Hans-Kristian.

Since this solution has already been mentioned in several other issues, I'm closing this one.

[memeruiz]

Another reason to get this error is to not have libnvidia-rtcore libnvoptix1 packages in debian (possibly also Ubuntu).

They give raytracing support to Nvidia in Linux