Azure: Fixed size vhd file necessary to create Azure VM images

[jomeier]

Hi,

currently I try to get OpenShift OKD working with the Azure fcos image.

The OKD installer downloads the xz compressed VHD file, decompresses it and uploads it into an Azure storage blob.

If we want to create an Azure VM image out of this VHD file Azure complains about that it is not fixed size but dynamic size.

Thats a problem because the common tools for converting VHD files from dynamic to fixed size are mostly written in Powershell or similar languages.

Is it possible that you offer fcos as a compressed fixed size VHD image so we don't have to convert it on our own which is not trivial but required for Azure ?

Because the image will be blown up mostly with zeroes, the compression should being able to take care about it.

Greetings,

Josef

[jomeier]

And if possible: offer it on the Azure marketplace, please. This would make things so much easier.

[LorbusChris]

xref: openshift/installer#3033 (comment)

Could we change coreos-assembler buildextend-azure to output a compressed fixed size VHD image that could be uploaded directly to Azure without any further steps?

cc'ing @dustymabe and @darkmuggle who I believe have worked in that cmd in the past.

[jlebon]

@jomeier Looks like you've already figured out the magic qemu-img convert flags to pass in openshift/installer#3033. Do you want to suggest it as a change to cosa? (Likely just needs a tweak in qemuvariants.py).

Would be good too to sanity check that the compressed image size indeed isn't affected much.

[lucab]

offer it on the Azure marketplace

This is tracked already at #148. Let's keep it out of this ticket here.

The OKD installer downloads the xz compressed VHD file, decompresses it and uploads it into an Azure storage blob.
If we want to create an Azure VM image out of this VHD file Azure complains about that it is not fixed size but dynamic size.

I feel like there is some piece that I'm missing.
To the best of my knowledge, Azure VHD images for FCOS and RHCOS are built the same way (but just differ in the final compression format). The latter seems to work with the blob we upload, but the former doesn't when the OKD installer uploads it.
Are we doing some hidden post-processing on RHCOS? Or is there some specific setting at upload time that we should document for FCOS users?

[LorbusChris]

This might be relevant here, too: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/prepare-for-upload-vhd-image

[jlebon]

Ahh, I think I see it. ore uses the azure-vhd-utils Go package, which conveniently converts dynamic to fixed on-the-fly: https://github.com/microsoft/azure-vhd-utils#how-upload-work. And again, RHCOS does uploads today, FCOS doesn't.

So presumably, the installer could vendor that package too and use that API. Not against just having it fixed in the first place in cosa though if the size difference is indeed negligeable.

[darkmuggle]

xref: openshift/installer#3033 (comment)

Could we change coreos-assembler buildextend-azure to output a compressed fixed size VHD image that could be uploaded directly to Azure without any further steps?

cc'ing @dustymabe and @darkmuggle who I believe have worked in that cmd in the past.

File an issue against COSA. We can look.

[LorbusChris]

Possibly relevant here, from https://docs.microsoft.com/en-us/azure/virtual-machines/windows/prepare-for-upload-vhd-image#convert-the-virtual-disk-to-a-fixed-size-and-to-vhd:

Regarding the size of the VHD:

All VHDs on Azure must have a virtual size aligned to 1MB. When converting from a raw disk to VHD you must ensure that the raw disk size is a multiple of 1 MB before conversion. Fractions of a megabyte will cause errors when creating images from the uploaded VHD.

[jomeier]

@jlebon:
You are right: azure-vhd-utils seems to do the conversion job. Currently I upload the decompressed vhd file and the storage blob is type page blob with 8GByte size.

The advantage of azure-vhd-utils is that it only transfers the data which is necessary. Because the conversion to fixed size means, that the VHD file is also expanded in size, efficiency plays a big role.

@LorbusChris: I made my first OKD tests on Azure with fcos and azure-vhd-utils. It does the job.

[jomeier]

Maybe it's absolutely sufficient to use azure-vhd-utils also with fcos in the fedora coreos ci/cd pipeline.

[lucab]

Not against just having it fixed in the first place in cosa though if the size difference is indeed negligeable.

I fetched the latest testing-devel image, converted to "fixed disktype" VHD (via qemu-img), and re-compressed (via xz):

• the final compressed size is only slightly larger (470MiB vs 435 MiB)
• intermediate uncompressed size is much larger (9.8 GiB vs 1.7 GiB)
• the compression step (xz -T 0) on my quadcore for "fixed" has run-time 1.5x than current (6 mins vs 4 mins)