> -----Original Message----- > From: Liu, Yi L <mailto:yi.l.liu@xxxxxxxxx> > Sent: Saturday, April 1, 2023 10:44 PM > Subject: [PATCH v3 00/12] Introduce new methods for verifying > ownership in vfio PCI hot reset > > VFIO_DEVICE_PCI_HOT_RESET requires user to pass an array of group fds > to prove that it owns all devices affected by resetting the calling > device. This series introduces several extensions to allow the > ownership check better aligned with iommufd and coming vfio device cdev support. > > First, resetting an unopened device is always safe given nobody is > using it. So relax the check to allow such devices not covered by > group fd array. [1] > > When iommufd is used we can simply verify that all affected devices > are bound to a same iommufd then no need for the user to provide extra > fd information. This is enabled by the user passing a zero-length fd > array and moving forward this should be the preferred way for hot > reset. [2] > > However the iommufd method has difficulty working with noiommu devices > since those devices don't have a valid iommufd, unless the noiommu > device is in a singleton dev_set hence no ownership check is required. > [3] > > For noiommu backward compatibility a 3rd method is introduced by > allowing the user to pass an array of device fds to prove ownership. > [4] > > As suggested by Jason [5], we have this series to introduce the above > stuffs to the vfio PCI hot reset. Per the dicussion in [6] [7], in the > end of this series, the VFIO_DEVICE_GET_PCI_HOT_RESET_INFO is extended > to report devid for the devices opened as cdev. This is goging to > support the device fd passing usage. > > The new hot reset method and updated _INFO ioctl are tested with two > test commits in below qemu: > > https://github.com/yiliu1765/qemu/commits/iommufd_rfcv3 > (requires to test with cdev kernel) Test pass with qemu supporting new uAPI, played trick to bypass FLR and trigger hot reset intentionally. 81:00.0 and 81:00.1 are dependent devices from same slot and passed to one VM, see below qemu trace for details. vfio_pci_hot_reset (0000:81:00.1) multi vfio_intx_disable_kvm (0000:81:00.1) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.1) vfio_pci_read_config (0000:81:00.1, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.1, @0x4, len=0x2) 0x142 vfio_pci_write_config (0000:81:00.1, @0x4, 0x140, len=0x2) vfio_pci_hot_reset_has_dep_devices 0000:81:00.1: hot reset dependent devices: vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.0 devid 2 vfio_intx_disable_kvm (0000:81:00.0) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.0) vfio_pci_read_config (0000:81:00.0, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.0, @0x4, len=0x2) 0x142 vfio_pci_write_config (0000:81:00.0, @0x4, 0x140, len=0x2) vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.1 devid 5 vfio_pci_hot_reset_result 0000:81:00.1 hot reset: Success vfio_pci_read_config (0000:81:00.0, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.0) KVM INTx accel enabled vfio_intx_enable (0000:81:00.0) vfio_pci_read_config (0000:81:00.1, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.1) KVM INTx accel enabled vfio_intx_enable (0000:81:00.1) vfio_pci_hot_reset (0000:81:00.0) multi vfio_intx_disable_kvm (0000:81:00.0) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.0) vfio_pci_read_config (0000:81:00.0, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.0, @0x4, len=0x2) 0x140 vfio_pci_write_config (0000:81:00.0, @0x4, 0x140, len=0x2) vfio_pci_hot_reset_has_dep_devices 0000:81:00.0: hot reset dependent devices: vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.0 devid 2 vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.1 devid 5 vfio_intx_disable_kvm (0000:81:00.1) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.1) vfio_pci_read_config (0000:81:00.1, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.1, @0x4, len=0x2) 0x140 vfio_pci_write_config (0000:81:00.1, @0x4, 0x140, len=0x2) vfio_pci_hot_reset_result 0000:81:00.0 hot reset: Success vfio_pci_read_config (0000:81:00.1, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.1) KVM INTx accel enabled vfio_intx_enable (0000:81:00.1) vfio_pci_read_config (0000:81:00.0, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.0) KVM INTx accel enabled vfio_intx_enable (0000:81:00.0) ...... vfio_pci_reset (0000:81:00.1) vfio_intx_disable_kvm (0000:81:00.1) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.1 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.1) vfio_pci_read_config (0000:81:00.1, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.1, @0x4, len=0x2) 0x140 vfio_pci_write_config (0000:81:00.1, @0x4, 0x140, len=0x2) vfio_pci_hot_reset (0000:81:00.1) one vfio_pci_hot_reset_has_dep_devices 0000:81:00.1: hot reset dependent devices: vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.0 devid 2 vfio_pci_read_config (0000:81:00.1, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.1) KVM INTx accel enabled vfio_intx_enable (0000:81:00.1) vfio_pci_reset (0000:81:00.0) vfio_intx_disable_kvm (0000:81:00.0) KVM INTx accel disabled vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 0 mmaps enabled: 1 vfio_region_mmaps_set_enabled Region 0000:81:00.0 BAR 3 mmaps enabled: 1 vfio_intx_disable (0000:81:00.0) vfio_pci_read_config (0000:81:00.0, @0x44, len=0x2) 0x2008 vfio_pci_read_config (0000:81:00.0, @0x4, len=0x2) 0x140 vfio_pci_write_config (0000:81:00.0, @0x4, 0x140, len=0x2) vfio_pci_hot_reset (0000:81:00.0) one vfio_pci_hot_reset_has_dep_devices 0000:81:00.0: hot reset dependent devices: vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.0 devid 2 vfio_pci_hot_reset_dep_devices_iommufd 0000:81:00.1 devid 5 vfio_pci_read_config (0000:81:00.0, @0x3d, len=0x1) 0x1 vfio_intx_enable_kvm (0000:81:00.0) KVM INTx accel enabled vfio_intx_enable (0000:81:00.0)
