mantle/platform/azure: Add support for Azure Shared Image Gallery (SIG) and other enhancements #4109

marmijo · 2025-05-17T00:34:40Z

This PR introduces support for building and running Azure Shared Image Gallery (SIG) images, also known as "gallery images", alongside traditional managed images. Both image types can now be used with either Hyper-V Generation 1 or Generation 2.

Additional enhancements include:

Support for overriding the instance type per kola test via external test configuration (useful for tests requiring specific VM sizes).
Ability to attach additional data disks to Azure instances, as specified in kola tests.

This enables testing the Azure disk udev rules that were added in coreos/fedora-coreos-config#3378, as NVMe support is only available on Gen2 Gallery Images.

See: https://issues.redhat.com/browse/COS-3125

commit 270e9e4c5cf175efe62381dc0bf57226c29a8586
Author: Michael Armijo <marmijo@redhat.com>
Date:   Fri May 16 17:11:02 2025 -0600

    mantle/kola: add InstanceType to PlatformOptions for external tests
    
    Add an `InstanceType` field  to `PlatformOptions` to allow external
    tests to override the instance type used in `kola run`. This is useful
    for cases where a specific test needs to run on a different (potentially
    more expensive) instance type. Support for this is currently limited
    to the Azure Platform.
    
    Also, fix the `MultiPathDisk` for the qemu-iso platform. The check is
    now correctly performed in the `NewMachineWithOptions` function, since
    `MultiPathDisk` is part of `platform.MachineOptions`.

commit 386e133a741e41a8969951b4f5a8ce5789ec72e6
Author: Michael Armijo <marmijo@redhat.com>
Date:   Wed May 14 17:58:40 2025 -0600

    ore/azure: add options to create and delete Shared Image Gallery Images
    
    Add a new `create-gallery-image` ore command to Support creating images
    within Azure Shared Image Galleries (Gallery Images). The command
    creates image definitions and versions in Azure Shared Image Galleries.
    Gallery images can be created from either a blob URL or an existing
    managed image. A `--azure-publisher` flag is added to assign a publisher
    to the gallery image.
    
    `delete-gallery-image` is also added to delete individual gallery images
    or an entire Shared Image Gallery.

commit 0be49d77209a89a8af65414d4ddec5cd099cb9e2
Author: Michael Armijo <marmijo@redhat.com>
Date:   Wed May 14 17:26:58 2025 -0600

    azure: add --azure-hyper-v-generation flag
    
    Add a new flag to allow specifying the Hyper-V generation (V1 or V2)
    when creating Azure images. This enables support for both Gen1 and Gen2
    image creation.


commit 54658f96aa716dc17f8da26a48d8cf08b3ef4f9c
Author: Michael Armijo <marmijo@redhat.com>
Date:   Wed May 14 16:49:34 2025 -0600

    platform/azure: support additional data disks
    
    Add support for attaching additional data disks to instances created in
    Azure. Disks are defined through the machines options as the Size in GB
    and the 'sku', or storage type, e.g. '["100G:sku=UltraSSD_LRS"]' for
    NVMe disks.

Introduce a new test verifies that udev rules[1] for Azure Managed NVMe disks correctly create symlinks in `/dev/disk/azure`. It only runs on Azure and uses new kola functionality[2] to override the instance type, enabling the use of a Hyper-V Gen2 VM (Standard_M16bds_v3) with a UltraSSD_LRS (NVMe) data disk attached. The test checks for the presence of expected symlinks: - /dev/disk/azure/os - /dev/disk/azure/data/by-lun/0 [1]: coreos#3378 [2]: coreos/coreos-assembler#4109

Introduce a new test which verifies that udev rules[1] for Azure Managed NVMe disks correctly create symlinks in `/dev/disk/azure`. It only runs on Azure and uses new kola functionality[2] to override the instance type, enabling the use of a Hyper-V Gen2 VM (Standard_M16bds_v3) with a UltraSSD_LRS (NVMe) data disk attached. The test checks for the presence of expected symlinks: - /dev/disk/azure/os - /dev/disk/azure/data/by-lun/0 [1]: coreos#3378 [2]: coreos/coreos-assembler#4109

Expand the job to build Hyper-V Gen2 gallery images and run kola tests on both traditional managed images and new Gen2 gallery images[1]. The new NVMe test[2] is explicitly denylisted from managed image testing since since it requires Gen2 support. This will essentially allow all Azure kola tests to be validated on both kinds of images. A pipeline config variable, `test_gallery`, is introduced to the Azure section to specify the gallery used to create Gen2 images. [1]: coreos/coreos-assembler#4109 [2]: coreos/fedora-coreos-config#3519

Introduce a new test which verifies that udev rules[1] for Azure Managed NVMe disks correctly create symlinks in `/dev/disk/azure`. It only runs on Azure and uses new kola functionality[2] to override the instance type, enabling the use of a Hyper-V Gen2 VM (Standard_M16bds_v3) with a UltraSSD_LRS (NVMe) data disk attached. The test checks for the presence of expected symlinks: - /dev/disk/azure/os - /dev/disk/azure/data/by-lun/0 [1]: coreos#3378 [2]: coreos/coreos-assembler#4109

marmijo · 2025-05-27T19:14:04Z

/retest rhcos

openshift-ci · 2025-05-27T19:14:07Z

@marmijo: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test images

/test rhcos

Use /test all to run all jobs.

In response to this:

/retest rhcos

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

marmijo · 2025-05-27T19:14:23Z

/test rhcos

Introduce a new test which verifies that udev rules[1] for Azure Managed NVMe disks correctly create symlinks in `/dev/disk/azure`. It only runs on Azure and uses new kola functionality[2] to override the instance type, enabling the use of a Hyper-V Gen2 VM (Standard_M16bds_v3) with a UltraSSD_LRS (NVMe) data disk attached. The test checks for the presence of OS and data disk symlinks, and that they point to a valid target. [1]: coreos#3378 [2]: coreos/coreos-assembler#4109

dustymabe · 2025-05-27T20:05:40Z

mantle/platform/api/azure/disk.go

+	return err
+}
+
+// example: ["10G:sku=UltraSSD_LRS"], the default disk type is "Standard_LRS"


The other functions describe what the function is doing too. Maybe we can do that in addition to the example?

dustymabe · 2025-05-27T20:09:53Z

mantle/platform/api/azure/instance.go

@@ -272,3 +286,45 @@ func (a *API) GetConsoleOutput(name, resourceGroup, storageAccount string) ([]by

 	return io.ReadAll(data)
 }
+
+func (a *API) AttachDiskToInstance(instanceName, resourceGroup, diskName string, sku armcompute.DiskStorageAccountTypes, sizeGB int32, lun int32) error {


This function name is already long, but might be nice to be more accurate. Suggestion: CreateAndAttachDiskToInstance().

dustymabe · 2025-05-27T20:19:27Z

mantle/cmd/ore/azure/azure.go

@@ -44,6 +45,7 @@ func init() {
 	sv := Azure.PersistentFlags().StringVar
 	sv(&azureCredentials, "azure-credentials", "", "Azure credentials file location (default \"~/"+auth.AzureCredentialsPath+"\")")
 	sv(&azureLocation, "azure-location", "westus", "Azure location (default \"westus\")")
+	sv(&azureHyperVGen, "azure-hyper-v-generation", "V1", "Azure Hypervisor Generation")


I would be happy not supporting an option here and just hardcoding v2 if that's achievable.

dustymabe · 2025-05-27T20:21:21Z

mantle/cmd/kola/options.go

@@ -101,6 +101,7 @@ func init() {
 	sv(&kola.AzureOptions.Location, "azure-location", "westus", "Azure location (default \"westus\"")
 	sv(&kola.AzureOptions.Size, "azure-size", "Standard_D2_v2", "Azure machine size (default \"Standard_D2_v2\")")
 	sv(&kola.AzureOptions.AvailabilityZone, "azure-availability-zone", "1", "Azure Availability Zone (default \"1\")")
+	sv(&kola.AzureOptions.HyperVGeneration, "azure-hyper-v-generation", "V1", "Azure Hyper-V Generation (default \"V1\")")


Should this be part of the previous commit?

dustymabe · 2025-05-27T20:23:40Z

mantle/cmd/ore/azure/create-gallery-image.go

+		Short:   "Create Azure Gallery image",
+		Long:    "Create Azure Gallery image from a blob url",
+		RunE:    runCreateGalleryImage,
+		Aliases: []string{"create-gallery-image-arm"},


Is having the alias useful or confusing?

dustymabe · 2025-05-27T20:37:53Z

mantle/platform/api/azure/gallery.go

+	}
+
+	// Create a Gallery Image Version
+	versionName := "1.0.0"


Definitely will want to update this when we start publishing images for other people to use.

dustymabe · 2025-05-27T20:40:37Z

mantle/platform/api/azure/gallery.go

+		versionPager := a.galImgVerClient.NewListByGalleryImagePager(resourceGroup, galleryName, imageName, nil)
+		for versionPager.More() {
+			versionPage, err := versionPager.NextPage(ctx)


In the future we'll definitely want to be able to delete a specific version of an image from an image gallery.

i.e. we'd call ore from cosa coreos-prune code to delete old versions of CoreOS we don't want to keep around any longer.

dustymabe · 2025-05-27T20:45:19Z

mantle/platform/api/azure/network.go

+	var ipProperties *armnetwork.PublicIPAddressPropertiesFormat
+	var ipZones []*string
+
+	// set SKU=Standard, Zllocation Method=Static and Availability Zone on public IPs when creating Gen2 images


Zllocation -> Allocation?

dustymabe · 2025-05-27T20:46:49Z

mantle/platform/api/azure/network.go

+			Name: to.Ptr(armnetwork.PublicIPAddressSKUNameStandard),
+		}
+		ipProperties = &armnetwork.PublicIPAddressPropertiesFormat{
+			PublicIPAllocationMethod: to.Ptr(armnetwork.IPAllocationMethodStatic),


Can't dynamically assign IPs on gen2?

dustymabe · 2025-05-27T20:52:16Z

mantle/platform/api/azure/instance.go

+	var size string
+	if opts.InstanceType != "" {
+		size = opts.InstanceType
+	} else {
+		size = a.opts.Size
+	}


I feel like this could use a comment.

We're basically overriding the kola set size with the one from the external test config?

openshift-ci · 2025-05-27T21:13:17Z

@marmijo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/rhcos	`a5edc64`	link	true	`/test rhcos`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Add support for attaching additional data disks to instances created in Azure. Disks are defined through the machines options as the Size in GB and the 'sku', or storage type, e.g. '["100G:sku=UltraSSD_LRS"]' for NVMe disks.

Add a new flag to allow specifying the Hyper-V generation (V1 or V2) when creating Azure images. This enables support for both Gen1 and Gen2 image creation.

Add a new `create-gallery-image` ore command to Support creating images within Azure Shared Image Galleries (Gallery Images). The command creates image definitions and versions in Azure Shared Image Galleries. Gallery images can be created from either a blob URL or an existing managed image. A `--azure-publisher` flag is added to assign a publisher to the gallery image. `delete-gallery-image` is also added to delete individual gallery images or an entire Shared Image Gallery.

Add an `InstanceType` field to `PlatformOptions` to allow external tests to override the instance type used in `kola run`. This is useful for cases where a specific test needs to run on a different (potentially more expensive) instance type. Support for this is currently limited to the Azure Platform. Also, fix the `MultiPathDisk` check for the qemu-iso platform. The check is now correctly performed in the `NewMachineWithOptions` function, since `MultiPathDisk` is part of `platform.MachineOptions`.

marmijo marked this pull request as draft May 17, 2025 01:10

openshift-ci bot added the do-not-merge/work-in-progress label May 17, 2025

marmijo force-pushed the azure-gen2-images branch from b76e3db to 8c1c513 Compare May 21, 2025 01:21

marmijo marked this pull request as ready for review May 21, 2025 23:25

openshift-ci bot removed the do-not-merge/work-in-progress label May 21, 2025

marmijo force-pushed the azure-gen2-images branch from 8c1c513 to 18626cb Compare May 21, 2025 23:33

marmijo mentioned this pull request May 22, 2025

tests/platforms: add test for Azure NVMe udev rules coreos/fedora-coreos-config#3519

Open

marmijo force-pushed the azure-gen2-images branch from 18626cb to 9131ca7 Compare May 22, 2025 17:05

marmijo mentioned this pull request May 22, 2025

jobs/kola-azure: run kola tests on both Managed and Gen2 Gallery Images coreos/fedora-coreos-pipeline#1153

Open

marmijo force-pushed the azure-gen2-images branch from 9131ca7 to a5edc64 Compare May 27, 2025 20:50

dustymabe reviewed May 27, 2025

View reviewed changes

dustymabe mentioned this pull request May 27, 2025

Uploading to cloud platforms: Azure coreos/fedora-coreos-tracker#148

Open

marmijo added 4 commits May 27, 2025 16:09

platform/azure: support additional data disks

8a7e586

Add support for attaching additional data disks to instances created in Azure. Disks are defined through the machines options as the Size in GB and the 'sku', or storage type, e.g. '["100G:sku=UltraSSD_LRS"]' for NVMe disks.

azure: add --azure-hyper-v-generation flag

2c2e59b

Add a new flag to allow specifying the Hyper-V generation (V1 or V2) when creating Azure images. This enables support for both Gen1 and Gen2 image creation.

marmijo force-pushed the azure-gen2-images branch from a5edc64 to 839442c Compare May 27, 2025 22:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

mantle/platform/azure: Add support for Azure Shared Image Gallery (SIG) and other enhancements #4109

mantle/platform/azure: Add support for Azure Shared Image Gallery (SIG) and other enhancements #4109

marmijo commented May 17, 2025 •

edited

Loading

Uh oh!

marmijo commented May 27, 2025

Uh oh!

openshift-ci bot commented May 27, 2025

Uh oh!

marmijo commented May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

dustymabe May 27, 2025

Uh oh!

openshift-ci bot commented May 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

mantle/platform/azure: Add support for Azure Shared Image Gallery (SIG) and other enhancements #4109

Are you sure you want to change the base?

mantle/platform/azure: Add support for Azure Shared Image Gallery (SIG) and other enhancements #4109

Conversation

marmijo commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marmijo commented May 27, 2025

Uh oh!

openshift-ci bot commented May 27, 2025

Uh oh!

marmijo commented May 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

marmijo commented May 17, 2025 •

edited

Loading

openshift-ci bot commented May 27, 2025 •

edited

Loading