On 21/08/2023 20:32, Laine Stump wrote: > A "VFIO variant" driver is a kernel driver for a device that supports > all the APIs of the basic vfio-pci driver (which enables assigning a > host PCI device to a QEMU guest) plus "some extra stuff" (e.g. to > enable things like saving/restoring device state in order to support > live migration.) > > Way back last year I posted a couple attempts to support VFIO variant > drivers; here is V2 (along with a later followup discussion from a > couple months ago): > > https://listman.redhat.com/archives/libvir-list/2022-August/233661.html > https://listman.redhat.com/archives/libvir-list/2023-May/240108.html > > The mlx5-vfio-pci driver has now been upstream for quite awhile (and > even in the downstream Fedora 38 kernel, for example), as are the > sysfs bits that allow us to determine whether or not a driver is a > VFIO variant, and I've updated the patch(es) to use this. > > I've also been working on auto-binding to the "best-match" VFIO > variant driver based on comparing the device's modalias file in sysfs > to the contents of the kernel's modules.alias file, but that isn't > quite ready (partly code that isn't yet working, but also partly > indecision about exactly where in the XML to put the driver name when > it is specified; I won't take up more space here with that though). > Don't we have a: <interface type='hostdev' managed='yes'> ... <driver name='vfio'> ... </interface/ ... that could be re-used somehow to specify the VFIO variant driver of choice? Somewhat similar to the nodedev-detach equivalent? OTOH, it would only cover networking hostdevs... :/ > In the meantime, there are people who want to use the mlx5-vfio-pci > driver (and Cedric Le Goater also has written vfio-pci-igbvf and > vfio-pci-e1000e drivers (which area very useful for testing), although > I don't think he has posted them anywhere yet), Interesting, e1000e and igbvf have quiesce/resume/dirty-tracking commands? Sounds too old to have :D unless of course this is Qemu-created commands and not actual IGB/E1000e hardware doing those > so I would like to get > the basic patches here merged in upstream now while I continue working > on "Part 2". > > These patches provide two improvements that make testing/using VFIO > drivers much more convenient: > > 1) The specific driver can be given in the virsh nodedev-detach > command (or the virNodeDeviceDetachFlags() API call), e.g.: > > virsh nodedev-detach pci_0000_04_11_5 --driver vfio-pci-igbvf > > 2) If the <hostdev> (or "<interface> ... <type='hostdev'/>" has > "managed='no'", then libvirt will recognize any VFIO variant driver > (rather than the current behavior of rejecting anything that isn't > exactly "vfio-pci") > > With these two capabilities, it's simple and straightforward to bind a > device to a VFIO variant driver, and then start a guest that uses that > device. > Nice Provided the initial use of VFIO variant drivers have been to support live migration, should the hostdev blocker be lifted too *just in case it is simple addition of course* ? In vfio 8.1 it is enabled by default, so there isn't much else to do other than unblock migration with hostdevs. Something like below is <interface> specific; which works but it is likely too specific: diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d21551ab07d9..6edc8583c539 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1316,13 +1316,15 @@ qemuMigrationSrcIsAllowedHostdev(const virDomainDef *def) continue; } - /* all other PCI hostdevs can't be migrated */ + /* some PCI hostdevs can't be migrated */ if (hostdev->parentnet) { - virDomainNetType actualType = virDomainNetGetActualType(hostdev->parentnet); - - virReportError(VIR_ERR_OPERATION_UNSUPPORTED, - _("cannot migrate a domain with <interface type='%1$s'>"), - virDomainNetTypeToString(actualType)); + /* + * Some VFIO network devices can be migrated if vendor + * VFIO driver is used e.g. via mlx5-vfio-pci. + * We leave it up to the live migration blocker in Qemu to + * stop us from proceeding. + */ + continue; } else { virReportError(VIR_ERR_OPERATION_UNSUPPORTED, _("cannot migrate a domain with <hostdev mode='subsystem' type='%1$s'>"), Or even something like this generic to the hostdev as long as it's assigned to a variant driver. For any off shot case that isn't a migration vfio driver then Qemu will fail for us as it will fail to probe the right things e.g. (below is totally untested, it is just for illustration purposes, we can probably do away with just VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VFIO and let Qemu block migration for us?) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d21551ab07d9..1f29493aa0c2 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1301,6 +1301,8 @@ qemuMigrationSrcIsAllowedHostdev(const virDomainDef *def) return false; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_PCI: + g_autoptr(virPCIDevice) pci = NULL; + /* * if the device has a <teaming type='transient'> * element, then migration *is* allowed because the @@ -1316,6 +1318,22 @@ qemuMigrationSrcIsAllowedHostdev(const virDomainDef *def) continue; } + if ((hostdev->source.subsys.u.pci.backend == VIR_DOMAIN_HOSTDEV_PCI_BACKEND_VFIO) && + !virHostdevGetPCIHostDevice(hostdev, &pci) && pci) { + g_autofree char *drvPath = NULL; + g_autofree char *drvName = NULL; + + /* + * Some VFIO network devices can be migrated if non standard + * VFIO driver is used. We leave it up to the live migration + * blocker in Qemu to stop us from proceeding. + */ + if (!virPCIDeviceGetCurrentDriverPathAndName(pci, &drvPath, + &drvName) && + !STREQ(drvName, "vfio-pci")) + continue; + } + /* all other PCI hostdevs can't be migrated */ if (hostdev->parentnet) { virDomainNetType actualType = virDomainNetGetActualType(hostdev->parentnet);
