Vulkan Drivers not available in serverless GPU container app

[robbbbbb]

Please provide us with the following information:

This issue is a: (mark with an x)

• bug report -> please search issues before submitting
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Issue description

ACA containers with serverless GPU support don't have vulkan drivers injected (by Nvidia Container Runtime) even when graphics driver capabilties are requested via NVIDIA_DRIVER_CAPABILITIES=all or NVIDIA_DRIVER_CAPABILITIES=graphics

This is preventing us from running rendering and visualisation workloads.

Steps to reproduce

1. Create an ACA app using the Microsoft sample image as per this tutorial https://learn.microsoft.com/en-us/azure/container-apps/gpu-image-generation?pivots=azure-portal
2. Add the NVIDIA_DRIVER_CAPABILITIES=all environment variable
3. Start or restart the app
4. Use the console to install vulkan-utils (apt update && apt-get -y install vulkan-tools)
5. Run nvidia-smi to confirm GPU is mounted
6. Run vulkaninfo to confirm Vulkan client drivers are working

Expected behavior [What you expected to happen.]
nvidia-smi enumerates T4 GPU
and
vulkaninfo displays information about the T4 GPU

Actual behavior [What actually happened.]
nvidia-smi enumerates the T4
but
vulkaninfo shows that no driver can be loaded

===========
VULKAN INFO
===========

Vulkan Instance Version: 1.1.70

Cannot create Vulkan instance.
/build/vulkan-UL09PJ/vulkan-1.1.70+dfsg1/demos/vulkaninfo.c:768: failed with VK_ERROR_INCOMPATIBLE_DRIVER

Additional context

Our ultimate aim is to deploy this as an app container job, but since apps and jobs share the same environment I assume whatever is preventing nvidia container runtime correctly injecting the vulkan drivers is the same in both cases.

[robbbbbb]

A small update possibly useful to anyone finding this issue prior to a fix being implemented at the Nvidia Container Runtime or host OS level: we managed to build an image as a stop gap which included the correct (at time of writing) vulkan drivers and ICD files by:

• Starting with Ubuntu 22.04
• Installing the user-space parts of the Nvidia drivers from the 550.54.15 installer

In docker terms that should look something like this:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y wget curl kmod
RUN wget https://in.download.nvidia.com/tesla/550.54.15/NVIDIA-Linux-x86_64-550.54.15.run
RUN chmod +x NVIDIA-Linux-x86_64-550.54.15.run
RUN ./NVIDIA-Linux-x86_64-550.54.15.run --no-kernel-module --no-questions --silent

Note: this is just a temporary workaround and will almost certainly break as soon as the ACA host OS Nvidia drivers change from version 550.54.15 as the userspace libraries will be incompatible with the kernel module.

[v-vish]

@robbbbbb We’ve labeled this issue as an enhancement request and will keep you updated as soon as we have any developments to share.

[Shane-oo]

Where can we track the progress of this 'enhancement'?. Vulkan is the modern way of gpu virtualization, it seems strange that gpu containers would be created without the necessary drivers for an application to run.