NVMe drive PCI passthrough and suprise hotplug

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


[AMD Official Use Only]


I am using Fedora 33, with the following KVM, qemu and libvirt versions:

QEMU 5.1.0

libvirt 6.6.0

KVM 5.14.18


We have done pass-through of a PCIe NVMe device to the guest running on FC33

using either virt-manager or virsh and then we do the hot-unplug of the device

while it is attached to the guest.


The device is no longer seen on the guest hardware device list on virt-manager

and then we hotplug the device again and we are able to use it on the Host,

but when we try to re-attach it to the guest, we get the following error message:


Requested operation is not valid, PCI device 0000:c4::00.0 is in use by driver QEMU,

Domain fedora 33.


So somehow libvirt still thinks the hot-unplugged device is attached.


Tracing the flow of hot un-plug event from guest to host :


->Guest pcie hotplug support detected the NVMe driver unplug (from guest kernel logs):


pciehp: Slot (0-6): Attention button pressed


pciehp: Slot (0-6): Powering off due to to button press.


-> Also looks like the guest notified Host/KVM (from host kernel logs):


pcieport: 0000:c4:0000.0: pciehp: Slot (208): Card not present


-> Correspondingly, vfio-pci module notified Qemu :


vfio-pci: 0000:c4:0000.0: Relaying device request to user (#0)


-> Then the un-plugged device reset is done.


vfio-pci: vfio_bar_restore: reset recovery - restoring BARs


pci 0000:c4:00.0: Removing from iommu group 105.


-> Next tried to verify if libvirt detected the DELETED_DRIVE event from qemu.


Running SystemTap script to capture events between qemu and libvirt :


stap examples/systemtap/qemu-monitor.stp


When the NVMe drive is attached to VM the following log output is seen from SystemTap:


execute "device-add", driver: "vfio-pci", host: "0000:c4:00.0", id: "hostdev0", bus: "pci.7", addr: "0".


When we hot-unplug the NVMe drive, the following log output is seen from SystemTap:


event: DEVICE_DELETED, device: "hostdev0", path: "/machine/peripheral/hostdev0".


So it looks like that qemu sent the "DEVICE_DELTED" event to libvirt, but libvirt has still not removed the attached

device from its bookeeping list.


I understand there is already a thread from 20202, discussing a similar issue :



But I am not sure if there is any fix/support added for this recently.


Looking for any feedback related to above and PCI device passthrough and hotplug support.




[Index of Archives]     [Virt Tools]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux