Update Azure default image

[cblmemo]

This PR updates the default Azure Image to ubuntu-hpc 20230803, an image with the latest NVIDIA driver(535.54.03) and CUDA version (12.2).

Reference: https://github.com/Azure/azhpc-images/releases/tag/ubuntu-hpc-20230803

For context, seems like all ubuntu-hpc image is V2 so I still use the original image for V1 instances.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
  • Except for several TPU tests, see TPU VM smoke test failed on latest master branch #2469
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[infwinston]

thanks @cblmemo! as this PR upgrades CUDA version to 12. we want to make sure it works for common GPUs. Can we test T4?

[infwinston]

what does this mean?

[cblmemo]

The gen_version is reference to HyperV generation IIUC?

skypilot/sky/clouds/service_catalog/data_fetchers/fetch_azure.py

Line 218 in 8125ebf

elif item['name'] == 'HyperVGenerations':

[Michaelvll]

The comment does not read. Could you rephrase the comment? Aslo, does the new ubuntu-hpc work for gen V1?

[cblmemo]

No, it only supports V2. To double-check, you can use:

import json
import subprocess
from concurrent.futures import ThreadPoolExecutor

images_output = subprocess.check_output(
    "az vm image list --publisher microsoft-dsvm --offer ubuntu-hpc --all", shell=True
).decode("utf-8")

images = json.loads(images_output)
print(f'Found {len(images)} images')


def check_image(image):
    output = (
        subprocess.check_output(
            (
                "az vm image show "
                f'--publisher {image["publisher"]} '
                f'--offer {image["offer"]} '
                f'--sku {image["sku"]} '
                f'--version {image["version"]} '
                '--query "hyperVGeneration" -o tsv'
            ),
            shell=True,
        )
        .decode("utf-8")
        .strip()
    )
    print(output)


with ThreadPoolExecutor(max_workers=10) as executor:
    list(executor.map(check_image, images))

[cblmemo]

thanks @cblmemo! as this PR upgrades CUDA version to 12. we want to make sure it works for common GPUs. Can we test T4?

I just realized there are not a lot of tests for Azure in our default smoke test... Will run with --azure flag again

[infwinston]

We could just test it using the TGI example. with T4 + 3b model? if this works we can merge it.

[Michaelvll]

Thanks for updating this @cblmemo! Left some comments about the cloud init.

[Michaelvll]

Do we really need to encode the script with base64?

[cblmemo]

Yes... Azure only accepts base64 encoded in the customData field. See #1910 (comment) for more information

[cblmemo]

Let me add a comment for this

[Michaelvll]

Same as above, do we really need to encode the cloud init to base64? Is it possible to use a plain text with some careful yaml syntax, e.g. using : | or : |- to avoid unexpected new line and indents?

[Michaelvll]

Thanks for the fix @cblmemo! Left several comments.

[Michaelvll]

The comment does not read. Could you rephrase the comment? Aslo, does the new ubuntu-hpc work for gen V1?

[Michaelvll]

Thanks for updating the image for Azure instances @cblmemo! cc'ing @romilbhardwaj to have a test on this PR for #2751.

[cblmemo]

Fixed - Please let me know when you finish testing @romilbhardwaj and I'll merge this 🫡

[Michaelvll]

The latest PR seems working with the following nvidia-smi output:

Thu Nov 16 23:26:09 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe          Off | 00000001:00:00.0 Off |                    0 |
| N/A   28C    P0              41W / 300W |      4MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |

Please feel free to merge it @cblmemo. : )

[datainvestor]

@cblmemo Is this only available on nightly? Because I keep getting error than the cuda driver is less than 11.8 when running Azure VM with A100 GPU

[concretevitamin]

@datainvestor Yes, this should be available in nightly: pip install -U skypilot-nightly