I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's
companion audio device through to a VM I am trying to start up.
I am trying this on two different hardware platforms, a HP zbook 15 and
a HP z800 workstation. Both systems are running Ubuntu 14.04.
Each time I try to start the VM on the laptop using "virsh start vmname"
I get the following errors:
error: Failed to start domain vmname
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
vfio: error opening /dev/vfio/1: Operation not permitted
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
vfio: failed to get group 1
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
Device initialization failed.
qemu-system-x86_64: -device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0:
Device 'vfio-pci' could not be initialized
If I start the VM from the command line on the z800 as root using the
qemu-system-x86_64 command directly it works. The VM starts and both of
the passed through devices show up in the VM. If I try to start the same
VM as the libvirt-qemu user using the same qemu-system-x86_64 command I
get the same above error (but with a different group number obviously).
My order of operations are:
blacklist the nouveau driver from loading (the closed source nvidia
driver is not loaded on either system)
In the kernel command line passed through grub on the z800 workstation I
have:
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
on the zbook15 laptop I have:
pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on
and on the laptop I have added the "allow_unsafe_interrupts=1" option to
the modprobe of that module.
On the z800 I manually bind the nvidia video and audio devices to the
pci-stub driver.
In both cases I see the following in the kernel dmesg:
(from the laptop)
[ 6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[ 6.342618] pci-stub 0000:01:00.0: claimed by stub
[ 6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000
[ 6.342632] pci-stub 0000:01:00.1: claimed by stub
(or from the z800 workstation)
[ 115.116860] pci-stub 0000:0f:00.1: claimed by stub
[ 157.126503] pci-stub 0000:0f:00.0: claimed by stub
I then use a vfio-bind script to bind both the video and audio devices
to the vfio driver.
Once I do that I see the appropriate files under /dev/vfio
(on the laptop)
drwxr-xr-x 2 root root 80 Nov 13 08:15 ./
drwxr-xr-x 18 root root 4380 Nov 13 08:14 ../
crw------- 1 root root 249, 1 Nov 13 08:15 1
crw-rw-rw- 1 root root 249, 0 Nov 13 08:14 vfio
(on the z800)
drwxr-xr-x 2 root root 80 Nov 13 10:26 ./
drwxr-xr-x 16 root root 4540 Nov 13 10:26 ../
crw------- 1 root root 247, 1 Nov 13 10:26 14
crw-rw-rw- 1 root root 247, 0 Nov 13 10:26 vfio
I have confirmed that the only devices in the iommu group is the nvidia
video and audio devices and I am attempting to pass both devices through
to the VM when I invoke it.
On the laptop I was seeing messages in the logs from apparmor each time
I tried to start the VM referring to a libvirt profile for this VM's
uuid. After looking into that for a while, I finally set the
security_driver in /etc/libvirt/qemu.conf to "none" and that stopped
those messages from showing up in the logs. This also prevented the
libvirt-UUID files from showing up for this VM in
/etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to
add the files in /dev/vfio to the apparmor profile and TEMPLATE files
but that did not seem to have any effect on the log messages.
In both cases I am actually wanting to start the VMs though openstack
which invokes the qemu-system-x86_64 process through libvirt. However if
I try to start the VM on the z800 workstation as root from the command
line using the following command:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp
2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device
ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1
-device
vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0
-device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive
file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device
ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd
-device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us
it works. If I try to start the VM on the z800 as the libvirt-qemu user
(user id virsh and openstack use) I get the error I listed at the top of
this E-mail. I have even tried changing the ownerships of the files in
/dev/vfio to this user but that did not make any difference and I still
get the same error.
I have spent quite a lot of time on this doing lots of research and I
seem to be no closer to solving the problem.
Can any experts on using VFIO see any problems with the steps and
configurations I have done or know of any other reason why I might be
getting this error sequence?
Any pointers to additional documentation or suggestions would be most
appreciated.
--
Steven DuChene
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
