On Thu, 2014-11-13 at 11:50 -0500, Steven DuChene wrote: > I am trying to use VFIO and pci-bind to pass a NVidia VGA card and it's > companion audio device through to a VM I am trying to start up. > > I am trying this on two different hardware platforms, a HP zbook 15 and > a HP z800 workstation. Both systems are running Ubuntu 14.04. > > Each time I try to start the VM on the laptop using "virsh start vmname" > I get the following errors: > > error: Failed to start domain vmname > error: internal error: early end of file from monitor: possible problem: > qemu-system-x86_64: -device > vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: > vfio: error opening /dev/vfio/1: Operation not permitted > qemu-system-x86_64: -device > vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: > vfio: failed to get group 1 > qemu-system-x86_64: -device > vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: > Device initialization failed. > qemu-system-x86_64: -device > vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0: > Device 'vfio-pci' could not be initialized > > If I start the VM from the command line on the z800 as root using the > qemu-system-x86_64 command directly it works. The VM starts and both of > the passed through devices show up in the VM. If I try to start the same > VM as the libvirt-qemu user using the same qemu-system-x86_64 command I > get the same above error (but with a different group number obviously). > > My order of operations are: > > blacklist the nouveau driver from loading (the closed source nvidia > driver is not loaded on either system) > > In the kernel command line passed through grub on the z800 workstation I > have: > > intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 > > on the zbook15 laptop I have: > > pci-stub.ids=10de:11fc,10de:0e0b intel_iommu=on > > and on the laptop I have added the "allow_unsafe_interrupts=1" option to > the modprobe of that module. > > On the z800 I manually bind the nvidia video and audio devices to the > pci-stub driver. > > In both cases I see the following in the kernel dmesg: > > (from the laptop) > [ 6.342603] pci-stub: add 10DE:11FC sub=FFFFFFFF:FFFFFFFF > cls=00000000/00000000 > [ 6.342618] pci-stub 0000:01:00.0: claimed by stub > [ 6.342625] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF > cls=00000000/00000000 > [ 6.342632] pci-stub 0000:01:00.1: claimed by stub > > (or from the z800 workstation) > [ 115.116860] pci-stub 0000:0f:00.1: claimed by stub > [ 157.126503] pci-stub 0000:0f:00.0: claimed by stub > > I then use a vfio-bind script to bind both the video and audio devices > to the vfio driver. > Once I do that I see the appropriate files under /dev/vfio > > (on the laptop) > drwxr-xr-x 2 root root 80 Nov 13 08:15 ./ > drwxr-xr-x 18 root root 4380 Nov 13 08:14 ../ > crw------- 1 root root 249, 1 Nov 13 08:15 1 > crw-rw-rw- 1 root root 249, 0 Nov 13 08:14 vfio > > (on the z800) > drwxr-xr-x 2 root root 80 Nov 13 10:26 ./ > drwxr-xr-x 16 root root 4540 Nov 13 10:26 ../ > crw------- 1 root root 247, 1 Nov 13 10:26 14 > crw-rw-rw- 1 root root 247, 0 Nov 13 10:26 vfio > > I have confirmed that the only devices in the iommu group is the nvidia > video and audio devices and I am attempting to pass both devices through > to the VM when I invoke it. > > On the laptop I was seeing messages in the logs from apparmor each time > I tried to start the VM referring to a libvirt profile for this VM's > uuid. After looking into that for a while, I finally set the > security_driver in /etc/libvirt/qemu.conf to "none" and that stopped > those messages from showing up in the logs. This also prevented the > libvirt-UUID files from showing up for this VM in > /etc/apparmor.d/libvirt/ each time I tried to start the VM. I did try to > add the files in /dev/vfio to the apparmor profile and TEMPLATE files > but that did not seem to have any effect on the log messages. > > In both cases I am actually wanting to start the VMs though openstack > which invokes the qemu-system-x86_64 process through libvirt. However if > I try to start the VM on the z800 workstation as root from the command > line using the following command: > > qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp > 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device > ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=sven.1 > -device > vfio-pci,host=01:00.0,bus=sven.1,addr=00.0,multifunction=on,x-vga=on,rombar=0 > -device vfio-pci,host=01:00.1,bus=sven.1,addr=00.1 -drive > file=/root/ubuntu_withdriver.raw,id=disk,format=raw -device > ide-hd,bus=ide.0,drive=disk -drive file=/root/my-seed.img,id=isocd > -device ide-cd,bus=ide.1,drive=isocd -vnc 0.0.0.0:1 -k en-us > > it works. If I try to start the VM on the z800 as the libvirt-qemu user > (user id virsh and openstack use) I get the error I listed at the top of > this E-mail. I have even tried changing the ownerships of the files in > /dev/vfio to this user but that did not make any difference and I still > get the same error. > > I have spent quite a lot of time on this doing lots of research and I > seem to be no closer to solving the problem. > > Can any experts on using VFIO see any problems with the steps and > configurations I have done or know of any other reason why I might be > getting this error sequence? What does your domain xml look for like assigning the device? I suspect you might be using a <qemu:args> block to define the assigned device so that you can add the x-vga=on option, which unfortunately has the side-effect that libvirt is no longer aware of the device and cannot set file permissions and locked memory limits correctly for doing device assignment. If so, you might be able to assign the audio function using a standard <hostdev> section, which should setup the permissions for the GPU in a <qemu:args> section as well since they're in the same group. Another option is to use a wrapper script around qemu, for instance: <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator> # cat /usr/libexec/qemu-kvm-nvidia-vga #!/bin/sh exec /usr/libexec/qemu-kvm \ `echo "\$@" | sed 's|01:00.0|01:00.0,x-vga=on|g'` You'll likely also need to update apparmor to allow libvirt to execute this script. For selinux you'd do something like described in this document: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Technical_Notes/virt.html The wrapper script option would allow both devices in the group to be fully managed by libvirt and would then remove your need to manually run any sort of vfio-bind script. However, I've now looked up the quickspecs for the zbook and see that you're likely using an Nvidia Quadro K-series GPU and hopefully something similar on the z800. These can be used as secondary GPUs in the VM and do not need the x-vga option. Simply attach the GPU and audio to the VM (using <hostdev>, not <qemu:arg>) in addition to one of the emulated VGA models. The emulated VGA will be used for boot and install and the assigned GPU will be activated when the Nvidia driver stack is loaded. Caveats: a) I have no idea if Ubuntu 14.04 includes the necessary support for GPU assignment. b) Laptop do strange and bizarre things with discrete GPUs supporting Optimus. It may not work, and if it does work, it may not output through any of the external video ports on the laptop or the screen. You might want to setup remote access to the VM to access the desktop via network and not rely on a physical output port. Good luck, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
