Enable GPU in Azure

[kevinxu-git]

Hello everyone,

I want to use the credit granted to me on Azure to train my model. I need for this to enable GPU.

I changed the type of instance to Standard NC6_Promo (6 vcpus, 56 GiB memory). But when I check if cuda is available with PyTorch or Keras, it returns False. I also tried to delete my Resource group and redeploy it by choosing GPU-56GB as type of instance, but the result is unchanged.

Does anyone know how to use the GPU on Azure for the challenge ?

Thank you in advance ! :)

[nickbutler]

Hi Kevin,

Thanks for reaching out!

According to the Azure documentation here additional drivers will need to be installed in order to utilise the GPU, which currently isn't handled by our template.

This can be done by modifying the deployment template before provisioning to add a VM extension (docs), the template can be modified in the browser by clicking the "Edit template" link at the top of the deployment parameters page. The VM extension could be added in at line 220, right after the extension used to install the ODC.

Alternatively, the drivers can be installed manually after the VM has been provisioned, as described here

We have not tested the use of GPUs for the challenge, so any further support we can provide is going to be limited, but please keep in touch with your progress.

Also, please be mindful of the costs associated with using the larger VM size. Check out the page on Cost Management.

We wish you the best of luck with the challenge!

[GasmElbary]

Hello!

I was wondering, did that work with you?
I've tried both ways, by modifying the deployment template before provisioning here, and by manually installing the drivers using (here). Unfortunately, both didn't fix the issue for me.

As for modifying the deployment template, I was able to deploy the resource environment successfully, but I couldn't access the Jupyter notebook using the public IP. Following the solution given in (here) for this issue, I was able to access the Jupyter notebook using the IP, but when I check for the available GPUs using a notebook I get False.

Adding the drivers manually was running smoothly, I even was able to see the available GPU when I use the nvidia_smi command. But I still receive False when I check for available GPUs using a notebook in the Jupyter notebook, which is weird.

Is there is a step I'm missing or anything else?

Thanks in advance :)

[nickbutler]

Hi @QM167,

It's great that you have tried this, but frustrating it hasn't worked, of course!

I haven't tested this myself yet. Upon reflection of my previous comment, it occurs to me that this may be because Jupyter is running in a Docker container, not on the physical host where the GPU is.

A brief googling and I came across Access an NVIDIA GPU in the official Docker docs, which is promising!

Here is the docker compose specific documentation

I will try this out as soon as I'm able, but if you have any success, please let us know!

Cheers

[GasmElbary]

Hi @codeindulgence ,

Thanks for your reply. Oo yes it is frustrating. The problem is that the model I want to train needs a GPU, CPU alone will not be enough.

What you said makes sense, but I'm facing trouble making the compose file. I installed the Docker Compose using (thiss) and the nvidia-container-runtime using (this). My problem is in creating the .yml file and running
docker-compose up

According to this I need to create the compose file with the mentioned details to enabling GPU access to service containers. I couldn't create a compose file and add text to it using the terminal. Instead, I created docker-compose.yml
and upload it online using my local device then download it using the terminal. But when I run the command I get the following error:

ubuntu@CubeInABox:~$ docker-compose up
ERROR: The Compose file is invalid because:
Service services have neither an image nor a build context specified. At least one must be provided.

Using the terminal I found the following images:

ubuntu@CubeInABox:~$ sudo docker images
REPOSITORY    TAG                             IMAGE ID       CREATED        SIZE
ubuntu        latest                          7e0aa2d69a15   5 weeks ago    72.7MB
nvidia/cuda   11.3.0-cudnn8-runtime-centos8   69abb6a125bd   5 weeks ago    3.45GB
hello-world   latest                          d1165f221234   2 months ago   13.3kB
nvidia/cuda   11.0-base                       2ec708416bb8   9 months ago   122MB
ubuntu@CubeInABox:~$

but IDK why no containers

ubuntu@CubeInABox:~$ sudo docker container ls
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
ubuntu@CubeInABox:~$

Just a question, how can I create a compose file using the terminal. Also, I believe the image to be used is the one called ubuntu, right? or it is nvidia/cuda?

Thank you again

[nickbutler]

Hey @QM167,

You can edit files in the terminal. The traditional terminal editor is vi/vim but it's not so friendly to new-comers.
The easier alternative is nano, e.g. run nano /opt/odc/docker-compose.yml. It displays the keyboard shortcuts at the bottom for how to save/close, etc.

If you have a working Jupyter on the VM, another option is to put the file in /opt/odc/notebooks and it should be available to edit in Jupyter.

The error you're seeing might be caused by an error in the YAML format or due to using a Windows file on a Linux OS.

Can you share your edited docker-compose.yml file here?
I'll see if I can spot any issues.

Cheers

[GasmElbary]

Hello @codeindulgence,

So, I've deleted the resource group from azure and did everything from the very beginning again.

I used to get this error when running sudo docker run -it --rm --gpus all ubuntu nvidia-smi

ubuntu@CubeInABox:~$ sudo docker run -it --rm --gpus all ubuntu nvidia-smi
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

but now I don't get it anymore. I get the expected result from here, which is

ubuntu@CubeInABox:~$ sudo docker run -it --rm --gpus all ubuntu nvidia-smi
Tue Jun  1 13:33:47 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA Tesla K80    On   | 00000001:00:00.0 Off |                    0 |
| N/A   64C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

the available images are:

ubuntu@CubeInABox:~$ sudo docker images
REPOSITORY                    TAG        IMAGE ID       CREATED         SIZE
geoscienceaustralia/sandbox   latest     d3ba426adbfc   11 days ago     4.21GB
ubuntu                        latest     7e0aa2d69a15   5 weeks ago     72.7MB
kartoza/postgis               11.0-2.5   00409934f62a   14 months ago   1.09GB

and the available containers are:

ubuntu@CubeInABox:~$ sudo docker container ls
CONTAINER ID   IMAGE                         COMMAND                  CREATED          STATUS          PORTS                                       NAMES
ee27df8cc6db   geoscienceaustralia/sandbox   "/bin/tini -s -- wit…"   53 minutes ago   Up 10 minutes   0.0.0.0:80->8888/tcp, :::80->8888/tcp       odc_jupyter_1
bb60710d711c   kartoza/postgis:11.0-2.5      "/bin/sh -c /docker-…"   53 minutes ago   Up 10 minutes   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   odc_postgres_1

I used nano editor to create the compose file. The content of the docker-compose.yml is

services:
  test:
    image: ubuntu
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: ["gpu"]

I tried with different images. When I put the image to ubunto or geoscienceaustralia/sandbox I get:

ubuntu@CubeInABox:~$ sudo docker-compose up
Recreating ubuntu_test_1 ... done
Attaching to ubuntu_test_1
test_1  | Tue Jun  1 13:35:38 2021
test_1  | +-----------------------------------------------------------------------------+
test_1  | | NVIDIA-SMI 465.19.01    Driver Version: 465.19.01    CUDA Version: 11.3     |
test_1  | |-------------------------------+----------------------+----------------------+
test_1  | | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
test_1  | | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
test_1  | |                               |                      |               MIG M. |
test_1  | |===============================+======================+======================|
test_1  | |   0  NVIDIA Tesla K80    On   | 00000001:00:00.0 Off |                    0 |
test_1  | | N/A   62C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |
test_1  | |                               |                      |                  N/A |
test_1  | +-------------------------------+----------------------+----------------------+
test_1  |
test_1  | +-----------------------------------------------------------------------------+
test_1  | | Processes:                                                                  |
test_1  | |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
test_1  | |        ID   ID                                                   Usage      |
test_1  | |=============================================================================|
test_1  | |  No running processes found                                                 |
test_1  | +-----------------------------------------------------------------------------+
ubuntu_test_1 exited with code 0

which is almost the same result from (here) but with the line Creating network "gpu_default" with the default driver missing in my case. I tried rebooting the VM and checking for available GPUs using a notebook in Jupyter Notebook, but I still get False. When I try image as kartoza/postgis or kartoza/postgis:11.0-2.5 I get the following and it stuck.

ubuntu@CubeInABox:~$ sudo docker-compose up
Recreating ubuntu_test_1 ... done
Attaching to ubuntu_test_1
test_1  | Add rule to pg_hba: 0.0.0.0/0
test_1  | Add rule to pg_hba: replication replicator
test_1  | Setup master database
test_1  | Creating Postgres data at /var/lib/postgresql/11/main
test_1  | Initializing Postgres Database at /var/lib/postgresql/11/main
test_1  | The files belonging to this database system will be owned by user "postgres".
test_1  | This user must also own the server process.
test_1  |
test_1  | The database cluster will be initialized with locales
test_1  |   COLLATE:  en_US.UTF-8
test_1  |   CTYPE:    en_US.UTF-8
test_1  |   MESSAGES: C
test_1  |   MONETARY: C
test_1  |   NUMERIC:  C
test_1  |   TIME:     C
test_1  | The default text search configuration will be set to "english".
test_1  |
test_1  | Data page checksums are disabled.
test_1  |
test_1  | fixing permissions on existing directory /var/lib/postgresql/11/main ... ok
test_1  | creating subdirectories ... ok
test_1  | selecting default max_connections ... 100
test_1  | selecting default shared_buffers ... 128MB
test_1  | selecting default timezone ... Etc/UTC
test_1  | selecting dynamic shared memory implementation ... posix
test_1  | creating configuration files ... ok
test_1  | running bootstrap script ... ok
test_1  | performing post-bootstrap initialization ... ok
test_1  | syncing data to disk ...
test_1  | WARNING: enabling "trust" authentication for local connections
test_1  | You can change this by editing pg_hba.conf or using the option -A, or
test_1  | --auth-local and --auth-host, the next time you run initdb.
test_1  | ok
test_1  |
test_1  | Success. You can now start the database server using:
test_1  |
test_1  |     /usr/lib/postgresql/11/bin/pg_ctl -D /var/lib/postgresql/11/main -l logfile start
test_1  |
test_1  | 2021-06-01 13:52:31.558 UTC [46] LOG:  listening on IPv4 address "127.0.0.1", port 5432
test_1  | psql: could not connect to server: No such file or directory
test_1  |       Is the server running locally and accepting
test_1  |       connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
test_1  | 2021-06-01 13:52:31.634 UTC [46] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
test_1  | 2021-06-01 13:52:31.727 UTC [58] LOG:  database system was shut down at 2021-06-01 13:52:12 UTC
test_1  | 2021-06-01 13:52:31.824 UTC [46] LOG:  database system is ready to accept connections
test_1  | 2021-06-01 13:52:31.824 UTC [64] FATAL:  database "gis" does not exist
test_1  | 2021-06-01 13:52:31.855 UTC [46] LOG:  background worker "pg_cron scheduler" (PID 64) exited with exit code 1
test_1  |                                   List of databases
test_1  |    Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
test_1  | -----------+----------+----------+-------------+-------------+-----------------------
test_1  |  postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
test_1  |  template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
test_1  |            |          |          |             |             | postgres=CTc/postgres
test_1  |  template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
test_1  |            |          |          |             |             | postgres=CTc/postgres
test_1  | (3 rows)
test_1  |
test_1  | postgres ready
test_1  | Setup postgres User:Password
test_1  | Creating superuser docker
test_1  | CREATE ROLE
test_1  | Creating replication user replicator
test_1  | 2021-06-01 13:52:32.856 UTC [121] FATAL:  database "gis" does not exist
test_1  | 2021-06-01 13:52:32.858 UTC [46] LOG:  background worker "pg_cron scheduler" (PID 121) exited with exit code 1
test_1  | CREATE ROLE
test_1  | Create db gis
test_1  | 2021-06-01 13:52:33.860 UTC [142] FATAL:  database "gis" does not exist
test_1  | 2021-06-01 13:52:33.862 UTC [46] LOG:  background worker "pg_cron scheduler" (PID 142) exited with exit code 1
test_1  | Enabling postgis in the database gis
test_1  | 2021-06-01 13:52:34.864 UTC [157] LOG:  pg_cron scheduler started
test_1  | CREATE EXTENSION
test_1  | Enabling hstore in the database gis
test_1  | CREATE EXTENSION
test_1  | Enabling postgis_topology in the database gis
test_1  | CREATE EXTENSION
test_1  | Loading legacy sql
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE FUNCTION
test_1  | CREATE OPERATOR CLASS
test_1  | ALTER DEFAULT PRIVILEGES
test_1  | CREATE EXTENSION
test_1  |                                   List of databases
test_1  |    Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
test_1  | -----------+----------+----------+-------------+-------------+-----------------------
test_1  |  gis       | docker   | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
test_1  |  postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
test_1  |  template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
test_1  |            |          |          |             |             | postgres=CTc/postgres
test_1  |  template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
test_1  |            |          |          |             |             | postgres=CTc/postgres
test_1  | (4 rows)
test_1  |
test_1  | 2021-06-01 13:52:36.587 UTC [46] LOG:  received smart shutdown request
test_1  | 2021-06-01 13:52:36.610 UTC [157] LOG:  pg_cron scheduler shutting down
test_1  | 2021-06-01 13:52:36.612 UTC [46] LOG:  background worker "logical replication launcher" (PID 65) exited with exit code 1
test_1  | 2021-06-01 13:52:36.612 UTC [59] LOG:  shutting down
test_1  | 2021-06-01 13:52:37.283 UTC [46] LOG:  database system is shut down
test_1  | Postgres initialisation process completed .... restarting in foreground
test_1  | 2021-06-01 13:52:37.632 UTC [245] LOG:  listening on IPv4 address "0.0.0.0", port 5432
test_1  | 2021-06-01 13:52:37.632 UTC [245] LOG:  listening on IPv6 address "::", port 5432
test_1  | 2021-06-01 13:52:37.679 UTC [245] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
test_1  | 2021-06-01 13:52:37.776 UTC [247] LOG:  database system was shut down at 2021-06-01 13:52:37 UTC
test_1  | 2021-06-01 13:52:37.803 UTC [245] LOG:  database system is ready to accept connections
test_1  | 2021-06-01 13:52:37.809 UTC [253] LOG:  pg_cron scheduler started

Any suggestions?

Thank you again