This series enhances the vfio-virtio driver to support live migration for virtio-net Virtual Functions (VFs) that are migration-capable. This series follows the Virtio 1.4 specification to implement the necessary device parts commands, enabling a device to participate in the live migration process. The key VFIO features implemented include: VFIO_MIGRATION_STOP_COPY, VFIO_MIGRATION_P2P, VFIO_MIGRATION_PRE_COPY. The implementation integrates with the VFIO subsystem via vfio_pci_core and incorporates Virtio-specific logic to handle the migration process. Migration functionality follows the definitions in uapi/vfio.h and uses the Virtio VF-to-PF admin queue command channel for executing the device parts related commands. Patch Overview: The first four patches focus on the Virtio layer and address the following: - Define the layout of the device parts commands required as part of the migration process. - Provide APIs to enable upper layers (e.g., VFIO, net) to execute the related device parts commands. The last three patches focus on the VFIO layer: - Extend the vfio-virtio driver to support live migration for Virtio-net VFs. - Move legacy I/O operations to a separate file, which is compiled only when VIRTIO_PCI_ADMIN_LEGACY is configured, ensuring that live migration depends solely on VIRTIO_PCI. Additional Notes: - The kernel protocol between the source and target devices includes a header containing metadata such as record size, tag, and flags. The record size allows the target to read a complete image from the source before passing device part data. This follows the Virtio specification, which mandates that partial device parts are not supplied. The tag and flags serve as placeholders for future extensions to the kernel protocol between the source and target, ensuring backward and forward compatibility. - Both the source and target comply with the Virtio specification by using a device part object with a unique ID during the migration process. As this resource is limited to a maximum of 255, its lifecycle is confined to periods when live migration is active. - According to the Virtio specification, a device has only two states: RUNNING and STOPPED. Consequently, certain VFIO transitions (e.g., RUNNING_P2P->STOP, STOP->RUNNING_P2P) are treated as no-ops. When transitioning to RUNNING_P2P, the device state is set to STOP and remains STOPPED until it transitions back from RUNNING_P2P->RUNNING, at which point it resumes its RUNNING state. - Furthermore, the Virtio specification does not support reading partial or incremental device contexts. This means that during the PRE_COPY state, the vfio-virtio driver reads the full device state. This step is beneficial because it allows the device to send some "initial data" before moving to the STOP_COPY state, thus reducing downtime by preparing early. To avoid an infinite number of device calls during PRE_COPY, the vfio-virtio driver limits this flow to a maximum of 128 calls. After reaching this limit, the driver will report zero bytes remaining in PRE_COPY, signaling to QEMU to transition to STOP_COPY. - Support for dirty page tracking during migration will be provided via the IOMMUFD framework. - This series has been successfully tested on Virtio-net VF devices. Yishai Yishai Hadas (7): virtio_pci: Introduce device parts access commands virtio: Extend the admin command to include the result size virtio: Manage device and driver capabilities via the admin commands virtio-pci: Introduce APIs to execute device parts admin commands vfio/virtio: Add support for the basic live migration functionality vfio/virtio: Add PRE_COPY support for live migration vfio/virtio: Enable live migration once VIRTIO_PCI was configured drivers/vfio/pci/virtio/Kconfig | 4 +- drivers/vfio/pci/virtio/Makefile | 3 +- drivers/vfio/pci/virtio/common.h | 127 +++ drivers/vfio/pci/virtio/legacy_io.c | 420 +++++++++ drivers/vfio/pci/virtio/main.c | 496 ++-------- drivers/vfio/pci/virtio/migrate.c | 1339 +++++++++++++++++++++++++++ drivers/virtio/virtio_pci_common.h | 19 +- drivers/virtio/virtio_pci_modern.c | 457 ++++++++- include/linux/virtio.h | 1 + include/linux/virtio_pci_admin.h | 11 + include/uapi/linux/virtio_pci.h | 131 +++ 11 files changed, 2593 insertions(+), 415 deletions(-) create mode 100644 drivers/vfio/pci/virtio/common.h create mode 100644 drivers/vfio/pci/virtio/legacy_io.c create mode 100644 drivers/vfio/pci/virtio/migrate.c -- 2.27.0
