Flaky manywheel builds : Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

[atalman]

See following errors in audio and vision :
https://github.com/pytorch/vision/actions/runs/5536861083/jobs/10105070853
https://github.com/pytorch/audio/actions/runs/5536859853/jobs/10105065270

Happens during initialize containers:

  Digest: sha256:98496e83272013c2c5a0d28a2759ad952372210559879f436bf57b529bd95b0f
  Status: Downloaded newer image for pytorch/manylinux-builder:cuda11.8
  docker.io/pytorch/manylinux-builder:cuda11.8
  /usr/bin/docker create --name 71f4513435284b16a1173e5b4f1e4775_pytorchmanylinuxbuildercuda118_9719a7 --label dc4f4d --workdir /__w/vision/vision --network github_network_bd81af612f9f40328280bfd2cbb98a6a --gpus all -e "HOME=/github/home" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/ec2-user/actions-runner/_work":"/__w" -v "/home/ec2-user/actions-runner/externals":"/__e":ro -v "/home/ec2-user/actions-runner/_work/_temp":"/__w/_temp" -v "/home/ec2-user/actions-runner/_work/_actions":"/__w/_actions" -v "/home/ec2-user/actions-runner/_work/_tool":"/__w/_tool" -v "/home/ec2-user/actions-runner/_work/_temp/_github_home":"/github/home" -v "/home/ec2-user/actions-runner/_work/_temp/_github_workflow":"/github/workflow" --entrypoint "tail" pytorch/manylinux-builder:cuda11.8 "-f" "/dev/null"
  6887acb3f5002bf674b558cae0b7e3dc00e57d85aac830fd4df03aee89e54adc
  /usr/bin/docker start 6887acb3f5002bf674b558cae0b7e3dc00e57d85aac830fd4df03aee89e54adc
  Error response from daemon: could not select device driver "" with capabilities: [[gpu]]
  Error: failed to start containers: 6887acb3f5002bf674b558cae0b7e3dc00e57d85aac830fd4df03aee89e54adc
  Error: Docker start fail with exit code 1

This error happens only with cuda builds.

Started happening in audio since Fri Jul 7, 12:16 pm:
https://github.com/pytorch/audio/actions/runs/5488475799/jobs/10001389130

This worklfows shows that, job may not always fail on rerun: https://github.com/pytorch/vision/actions/runs/5519379435/jobs/10065684909

[malfet]

I wonder if we can print some additional information in docker version...

[malfet]

I can easily reproduce this error while trying to passhru GPU's on a CPU-only machine:

$ docker run --gpus all --rm -it pytorch/manylinux-builder:cpu bash
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

[atalman]

I suspect issue is that we run LInux build jobs like this:

    container:
      image: ${{ matrix.container_image }}
    # If a build is taking longer than 60 minutes on these runners we need
    # to have a conversation

The host is GPU host, but it maybe missing nvidia-smi driver.
We don't have this issue in core since we preinstall nvidia-smi driver, like this:

and only then we load the Docker image.

We have following options:

1. Switch builds to use separate Docker Pulll/Docker execute steps rather then GHA provided container functionality.
2. Keep build steps on the CPU machines, move validation workflows to Nova and use validation workflows for smoke testing. basically switching test to GPU machines

[atalman]

cc @malfet I think we lost nvidia-smi installation in docker from this commit:
pytorch/builder@bcd1f7b#diff-c56082e799a7ccf35a475e9056aaf83c076afd6f1f29c84e8ab68a4ec6e2962aR36

[malfet]

@atalman why do we need nvidia-smi inside the container to be able to access CUDA?
Also, isn't it provided by docker runtime:

$ docker run --gpus all --rm -it pytorch/manylinux-builder:cpu which nvidia-smi
/usr/bin/nvidia-smi
$ docker run  --rm -it pytorch/manylinux-builder:cpu which nvidia-smi
which: no nvidia-smi in (/opt/rh/devtoolset-9/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)