Thanks Alex ,
That works for me
~$ sudo driverctl --nosave set-override 0000:00:05.0 vfio-pci
~$ sudo driverctl --nosave set-override 0000:00:06.0 vfio-pci
~$ sudo driverctl --nosave set-override 0000:00:07.0 vfio-pci
admin@node-1:~$ sudo nvme list
Node SN Model
Namespace Usage Format FW
Rev
--------------------- --------------------
---------------------------------------- ---------
-------------------------- ---------------- --------
/dev/nvme10n1 akqvf2-0_10 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme11n1 akqvf2-0_11 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme5n1 akqvf2-0_5 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme6n1 akqvf2-0_6 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme7n1 akqvf2-0_7 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme8n1 akqvf2-0_8 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
/dev/nvme9n1 akqvf2-0_9 QEMU NVMe Ctrl
1 274.88 GB / 274.88 GB 512 B + 0 B 1.0
I came across you blogspot after I fired the mail
http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html
Some should update https://docs.kernel.org/driver-api/vfio.html in
public interest , If I knew how to do that I would do it,
Thanks again.
On Thu, Dec 8, 2022 at 10:55 PM Alex Williamson
<alex.williamson@xxxxxxxxxx> wrote:
>
> On Thu, 8 Dec 2022 10:50:08 -0600
> Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
>
> > [+cc VFIO folks and Zhenzhong (author of the commit you mention)]
> >
> > On Thu, Dec 08, 2022 at 09:24:31PM +0530, Major Saheb wrote:
> > > I have a linux system running in kvm, with 6 qemu emulated NVMe
> > > drives, as expected all of them have the same PCIe Vendor ID and
> > > Device ID(VID: 0x1b36 DID: 0x0010).
> > >
> > > When I try to unbind them from the kernel NVMe driver and bind it to
> > > vfio-pci one by one, I am getting "write error: File exists" when I
> > > try to bind the 2nd(and other) drive to vfio-pci.
> > >
> > > Kernel version
> > >
> > > 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
> > > x86_64 x86_64 GNU/Linux
> > >
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme0n1 -> ../devices/pci0000:00/0000:00:03.0/nvme/nvme0/nvme0n1
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme1n1 -> ../devices/pci0000:00/0000:00:04.0/nvme/nvme1/nvme1n1
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme2n1 -> ../devices/pci0000:00/0000:00:05.0/nvme/nvme2/nvme2n1
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme3n1 -> ../devices/pci0000:00/0000:00:06.0/nvme/nvme3/nvme3n1
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme4n1 -> ../devices/pci0000:00/0000:00:07.0/nvme/nvme4/nvme4n1
> > > lrwxrwxrwx 1 root root 0 Dec 8 11:32 /sys/block/nvme5n1 -> ../devices/pci0000:00/0000:00:08.0/nvme/nvme5/nvme5n1
> > >
> > > Steps for repro
> > > ubind nvme2 from kernel NVMe driver and bind it to vfio
> > > $ ls -l /sys/bus/pci/drivers/vfio-pci/
> > > lrwxrwxrwx 1 root root 0 Dec 8 13:04 0000:00:05.0 -> ../../../../devices/pci0000:00/0000:00:05.0
> > > --w------- 1 root root 4096 Dec 8 13:07 bind
> > > lrwxrwxrwx 1 root root 0 Dec 8 13:07 module -> ../../../../module/vfio_pci
> > > --w------- 1 root root 4096 Dec 8 13:04 new_id
> > > --w------- 1 root root 4096 Dec 8 13:07 remove_id
> > > --w------- 1 root root 4096 Dec 8 11:32 uevent
> > > --w------- 1 root root 4096 Dec 8 13:07 unbind
> > >
> > > Unbind nvme3 from kernel NVMe driver
> > > Try binding to vfio-pci
> > > # echo "0x1b36 0x0010" > /sys/bus/pci/drivers/vfio-pci/new_id
> > > -bash: echo: write error: File exists
>
> Presumably you already wrote this same ID to the dynamic ID table from
> the first device, so yes, it's going to rightfully complain that this
> ID already exists. The new_id interface has numerous problems, which
> is why we added the driver_override interface, which is used by tools
> like libvirt and driverctl in place of this old interface.
>
> I'd recommend something like:
>
> # driverctl --nosave set-override 0000:00:03.0 vfio-pci
> # driverctl --nosave set-override 0000:00:04.0 vfio-pci
> # driverctl --nosave set-override 0000:00:05.0 vfio-pci
> ...
>
> Or if vfio-pci is generally the preferred driver for these devices, you
> could remove the --nosave option to have them automatically bound at
> boot. You could also make use of pre-filling the vfio device table
> using vfio-pci.ids=1b36:0010 on the kernel command line and making sure
> the vfio-pci driver is loaded before the nvme driver. In general, for
> dynamic binding of devices, driver_override is the recommended solution.
> Thanks,
>
> Alex
>
