Consider adding VFIO_USER_GET_PRECOPY_INFO

[eltrufas]

I'm taking a shot at implementing VFIO migration v2 support for the QEMU vfio-user client.

One thing I ran into is that QEMU relies on the VFIO_MIG_GET_PRECOPY_INFO ioctl to determine when to transition the device from PRE_COPY to STOP_COPY. vfio-user doesn't seem to expose partial progress of a migration currently, so it'd be useful to have an equivalent of the VFIO ioctl.

If this seems like a bad idea and there's a better way to manage the state transition in QEMU, I'd love to hear it (this is my first foray into hacking on QEMU so please bear with me if I'm missing something obvious).

[eltrufas]

I'd be happy to work on contributing a patch for this but I'd like to defer an API design decision about the migration callbacks. Since the VFIO API distinguishes between initial and dirty remaining bytes, should libvfio-user:

• Request both dirty and initial in the callback, so the server implementor has to keep track of that.
• Keep track of the initial value internally (getting the remaining bytes with the callback on transition to PRE_COPY). This way, server implementations only need to keep track the total remaining migration data. Or,
• Break from the VFIO API and only return a total remaining bytes number (my least favorite).

[jlevon]

I'm taking a shot at implementing VFIO migration v2 support for the QEMU vfio-user client.

Wow, excellent! Do you have a particular implementation in mind? We previously focused on vNVMe in SPDK, but perhaps you have something else.

In general, we should aim to keep the end user implementation simple, so if it makes sense to you, keeping track in the library sounds like the preferred approach.

[tmakatos]

One thing I ran into is that QEMU relies on the VFIO_MIG_GET_PRECOPY_INFO ioctl to determine when to transition the device from PRE_COPY to STOP_COPY. vfio-user doesn't seem to expose partial progress of a migration currently, so it'd be useful to have an equivalent of the VFIO ioctl.

FWIW PRE_COPY is optional and IIUC QEMU handles that (https://github.com/qemu/qemu/blob/master/hw/vfio/migration.c#L475), so strictly speaking it's not necessary to support it, but it is nice to have it for sure.

Keeping track of the initial_bytes value internally definitely sounds like better from the user's perspective, however I can't help wondering if it's as simple as that then why wasn't it done this way in the kernel in the first place? I'm also (still) reading the QEMU VFIO client migration code it's not obvious to me that initial_bytes cannot change. Would you be willing to dig in further to asnwer those questions?

[eltrufas]

Wow, excellent! Do you have a particular implementation in mind? We previously focused on vNVMe in SPDK, but perhaps you have something else.

SPDK's vfio targets (nvme mainly) are my ultimate goal as well. I'm only testing with the GPIO sample so far, though.

Would you be willing to dig in further to asnwer those questions?

I hadn't thought of those points but I've been giving this some thought, and though I don't really know the intention of the VFIO API in that regard, I have some ideas.

I'm also (still) reading the QEMU VFIO client migration code it's not obvious to me that initial_bytes cannot change.

It's expected that initial_bytes monotonically decreases every time precopy info is queried as data is read out. My first idea for the implementation was to record the pending bytes at the start of the precopy phase and then just decrease that every time a MIG_DATA_READ completes.

I see now that that assumes drivers will always order the data read out so that you cannot send out data that's not in the set captured by initial_bytes. This probably holds true for most backends, but it's probably best to keep the flexibility to allow drivers to do more complicated progress tracking. It should be fine since the simple case that most servers fall into is fairly simple to implement (record size on transition to PRE_COPY, and reduce that number on every read).

Interestingly (to me at least), qemu does make the assumption I mentioned above when it's estimating precopy progress. It does later on do VFIO_GET_PRECOPY_INFO when it gets the exact pending number.

I have some draft patches for QEMU and libvfio-user to enable migration support, but I sadly haven't even been able get to the iteration part because there seems to be some problem with the DMA logging parts that I'm really struggling to understand so far. I'll open a separate issue for that once I've gotten a better handle on the problem and I've uploaded my tree.

[eltrufas]

FWIW PRE_COPY is optional and IIUC QEMU handles that (https://github.com/qemu/qemu/blob/master/hw/vfio/migration.c#L475), so strictly speaking it's not necessary to support it, but it is nice to have it for sure.

To add to this, even if the precopy info request isn't strictly required, it seems like even in STOP_COPY, QEMU relies on the VFIO_DEVICE_FEATURE_MIG_DATA_SIZE feature (https://github.com/qemu/qemu/blob/master/hw/vfio/migration.c#L312). So at least that will be necessary.

[tmakatos]

FWIW PRE_COPY is optional and IIUC QEMU handles that (https://github.com/qemu/qemu/blob/master/hw/vfio/migration.c#L475), so strictly speaking it's not necessary to support it, but it is nice to have it for sure.

To add to this, even if the precopy info request isn't strictly required,

I didn't mean to say that if it's not strictly required then we sholdn't support it, but merely pointing out that we can probably skip it for now to get the basic migration working. I'll try to get some clarity on the exact semantics of initial_bytes.

it seems like even in STOP_COPY, QEMU relies on the VFIO_DEVICE_FEATURE_MIG_DATA_SIZE feature (https://github.com/qemu/qemu/blob/master/hw/vfio/migration.c#L312). So at least that will be necessary.

Yes VFIO_DEVICE_FEATURE_MIG_DATA_SIZE does seem necessary, it comes from here:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4e016f969529f2aec0545e90119e7eb3cb124c46

Strange we missed in the original work to support migration v2. Are you planning to work on this as well?

[jlevon]

Hi @eltrufas , did you manage to make any progress on live migration in general?