Skip to content

Design proposal for supported image formats (v3) #6550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 25, 2025
Merged

Design proposal for supported image formats (v3) #6550

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 114 additions & 24 deletions doc/content/design/sm-supported-image-formats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Add supported image formats in sm-list
layout: default
design_doc: true
revision: 2
revision: 3
status: proposed
---

Expand All @@ -22,32 +22,16 @@ available formats.
# Design Proposal

To expose the available image formats to clients (e.g., XenCenter, XenOrchestra, etc.),
we propose adding a new field called `supported-image-formats` to the Storage Manager (SM)
module. This field will be included in the output of the `SM.get_all_records` call.
we propose adding a new field called `supported_image_formats` to the Storage Manager
(SM) module. This field will be included in the output of the `SM.get_all_records` call.

The `supported-image-formats` field will be populated by retrieving information
from the SMAPI drivers. Specifically, each driver will update its `DRIVER_INFO`
dictionary with a new key, `supported_image_formats`, which will contain a list
of strings representing the supported image formats
(for example: `["vhd", "raw", "qcow2"]`).

The list designates the driver's preferred VDI format as its first entry. That
means that when migrating a VDI, the destination storage repository will
attempt to create a VDI in this preferred format. If the default format cannot
be used (e.g., due to size limitations), an error will be generated.

If a driver does not provide this information (as is currently the case with existing
drivers), the default value will be an empty array. This signifies that it is the
driver that decides which format it will use. This ensures that the modification
remains compatible with both current and future drivers.

With this new information, listing all parameters of the SM object will return:
- With this new information, listing all parameters of the SM object will return:

```bash
# xe sm-list params=all
```

will output something like:
Output of the command will look like (notice that CLI uses hyphens):

```
uuid ( RO) : c6ae9a43-fff6-e482-42a9-8c3f8c533e36
Expand All @@ -65,12 +49,118 @@ required-cluster-stack ( RO) :
supported-image-formats ( RO) : vhd, raw, qcow2
```

This change impacts the SM data model, and as such, the XAPI database version will
be incremented.
## Implementation details

The `supported_image_formats` field will be populated by retrieving information
from the SMAPI drivers. Specifically, each driver will update its `DRIVER_INFO`
dictionary with a new key, `supported_image_formats`, which will contain a list
of strings representing the supported image formats
(for example: `["vhd", "raw", "qcow2"]`). Although the formats are listed as a
list of strings, they are treated as a set-specifying the same format multiple
times has no effect.

### Driver behavior without `supported_image_formats`

If a driver does not provide this information (as is currently the case with
existing drivers), the default value will be an empty list. This signifies
that the driver determines which format to use when creating VDI. During a migration,
the destination driver will choose the format of the VDI if none is explicitly
specified. This ensures backward compatibility with both current and future drivers.

### Specifying image formats for VDIs creation

If the supported image format is exposed to the client, then, when creating new VDI,
user can specify the desired format via the `sm_config` parameter `image-format=qcow2` (or
any format that is supported). If no format is specified, the driver will use its
preferred default format. If the specified format is not supported, an error will be
generated indicating that the SR does not support it. Here is how it can be achieved
using the XE CLI:

```bash
# xe vdi-create \
sr-uuid=cbe2851e-9f9b-f310-9bca-254c1cf3edd8 \
name-label="A new VDI" \
virtual-size=10240 \
sm-config:image-format=vhd
```

### Specifying image formats for VDIs migration

When migrating a VDI, an API client may need to specify the desired image format if
the destination SR supports multiple storage formats.

#### VDI pool migrate

To support this, a new parameter, `dest_img_format`, is introduced to
`VDI.pool_migrate`. This field accepts a string specifying the desired format (e.g., *qcow2*),
ensuring that the VDI is migrated in the correct format. The new signature of
`VDI.pool_migrate` will be
`VDI ref pool_migrate (session ref, VDI ref, SR ref, string, (string -> string) map)`.

If the specified format is not supported or cannot be used (e.g., due to size limitations),
an error will be generated. Validation will be performed as early as possible to prevent
disruptions during migration. These checks can be performed by examining the XAPI database
to determine whether the SR provided as the destination has a corresponding SM object with
the expected format. If this is not the case, a `format not found` error will be returned.
If no format is specified by the client, the destination driver will determine the appropriate
format.

```bash
# xe vdi-pool-migrate \
uuid=<VDI_UUID> \
sr-uuid=<SR_UUID> \
dest-img-format=qcow2
```

#### VM migration to remote host

A VDI migration can also occur during a VM migration. In this case, we need to
be able to specify the expected destination format as well. Unlike `VDI.pool_migrate`,
which applies to a single VDI, VM migration may involve multiple VDIs.
The current signature of `VM.migrate_send` is `(session ref, VM ref, (string -> string) map,
bool, (VDI ref -> SR ref) map, (VIF ref -> network ref) map, (string -> string) map,
(VGPU ref -> GPU_group ref) map)`. Thus there is already a parameter that maps each source
VDI to its destination SR. We propose to add a new parameter that allows specifying the
desired destination format for a given source VDI: `(VDI ref -> string)`. It is
similar to the VDI-to-SR mapping. We will update the XE cli to support this new format.
It would be `image_format:<source-vdi-uuid>=<destination-image-format>`:

```bash
# xe vm-migrate \
host-uuid=<HOST_UUID> \
remote-master=<IP> \
remote-password=<PASS> \
remote-username=<USER> \
vdi:<VDI1_UUID>=<SR1_DEST_UUID> \
vdi:<VDI2_UUID>=<SR2_DEST_UUID> \
image-format:<VDI1_UUID>=vhd \
image-format:<VDI2_UUID>=qcow2 \
uuid=<VM_UUID>
```
The destination image format would be a string such as *vhd*, *qcow2*, or another
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architecturally, the destination format could be a list of formats in order of preference where we currently keep this to one element. But maybe this is overloading this feature and we could do this later as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we could do it later but I agree that it is a good idea.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are talking about lists, sets, and so on: we need to be careful if those exist as a data structure on the API level or are represented as strings somewhere else. For a string representation details matter: is this a space separated or comma separated? The formats by themselves are probably words that must not contain punctuation or spaces.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly only the "supported image format" provided by the SM plugin will be stored in the xapi database. And it will be a list of image format like the one find for capabilities
So in DRIVER_INFO you will have a new field supported_image_format with the list of supported image format (that will be a set of strings).

DRIVER_INFO = {
    'name': 'Local EXT3 VHD',
    'description': 'SR plugin which represents disks as VHD files stored on a local EXT3 filesystem, created inside an LVM volume',
    ...
    'supported_image_formats': ['vhd', 'raw']
    }

So yes formats will be words without punctuation or spaces. And I didn't mention it (I should) but it will be case insensitive. "qcow2" is equivalent to "Qcow2".

supported format. It is optional to specify a format. If omitted, the driver
managing the destination SR will determine the appropriate format.
As with VDI pool migration, if this parameter is not supported by the SM driver,
a `format not found` error will be returned. The validation must happen before
sending a creation message to the SM driver, ideally at the same time as checking
whether all VDIs can be migrated.

To be able to check the format, we will need to modify `VM.assert_can_migrate` and
add the mapping from VDI references to their image formats, as is done in `VM.migrate_send`.

# Impact

- **Data Model:** A new field (`supported-image-formats`) is added to the SM records.
It should have no impact on existing storage repositories that do not provide any information
about the supported image format.

This change impacts the SM data model, and as such, the XAPI database version will
be incremented. It also impacts the API.

- **Data Model:**
- A new field (`supported_image_formats`) is added to the SM records.
- A new parameter is added to `VM.migrate_send`: `(VDI ref -> string) map`
- A new parameter is added to `VM.assert_can_migrate`: `(VDI ref -> string) map`
- A new parameter is added to `VDI.pool_migrate`: `string`
- **Client Awareness:** Clients like the `xe` CLI will now be able to query and display the supported image formats for a given SR.
- **Database Versioning:** The XAPI database version will be updated to reflect this change.

Loading