On Thu, 8 Dec 2022 10:50:08 -0600 Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > [+cc VFIO folks and Zhenzhong (author of the commit you mention)] > > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote: > > I have a linux system running in kvm, with 6 qemu emulated NVMe > > drives, as expected all of them have the same PCIe Vendor ID and > > Device ID(VID: 0x1b36 DID: 0x0010). > > > > When I try to unbind them from the kernel NVMe driver and bind it to > > vfio-pci one by one, I am getting "write error: File exists" when I > > try to bind the 2nd(and other) drive to vfio-pci. > > > > Kernel version > > > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 > > x86_64 x86_64 GNU/Linux > > > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1 > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1 > > > > Steps for repro > > ubind nvme2 from kernel NVMe driver and bind it to vfio > > $ ls -l /sys/bus/pci/drivers/vfio-pci/ > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0 > > --w------- 1 root root 4096 Dec 8 13:07 bind > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci > > --w------- 1 root root 4096 Dec 8 13:04 new_id > > --w------- 1 root root 4096 Dec 8 13:07 remove_id > > --w------- 1 root root 4096 Dec 8 11:32 uevent > > --w------- 1 root root 4096 Dec 8 13:07 unbind > > > > Unbind nvme3 from kernel NVMe driver > > Try binding to vfio-pci > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id > > -bash: echo: write error: File exists Presumably you already wrote this same ID to the dynamic ID table from the first device, so yes, it's going to rightfully complain that this ID already exists. The new_id interface has numerous problems, which is why we added the driver_override interface, which is used by tools like libvirt and driverctl in place of this old interface. I'd recommend something like: # driverctl --nosave set-override 0000:00:03.0 vfio-pci # driverctl --nosave set-override 0000:00:04.0 vfio-pci # driverctl --nosave set-override 0000:00:05.0 vfio-pci ... Or if vfio-pci is generally the preferred driver for these devices, you could remove the --nosave option to have them automatically bound at boot. You could also make use of pre-filling the vfio device table using vfio-pci.ids=1b36:0010 on the kernel command line and making sure the vfio-pci driver is loaded before the nvme driver. In general, for dynamic binding of devices, driver_override is the recommended solution. Thanks, Alex
