> Sent: Friday, June 21, 2024 at 12:15 PM > From: "daggs via Users" <users@xxxxxxxxxxxxxxxxx> > To: users@xxxxxxxxxxxxxxxxx > Subject: unable to start a nested vm due to iommu_group issue > > Greetings, > > I'm working on a new os for my server which runs 2 vms. I'm using nested vms to work on it so I won't take down the server. > the new os is alpine 3.20, qemu is 9.0.1, libvirt is 10.3.0. > > I have a device I want to pass to one of the guests, here is the iommu group layout: > IOMMU Group 18 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02) > IOMMU Group 18 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02) > IOMMU Group 18 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02) > IOMMU Group 18 00:1f.4 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) > > the device in question is 00:1f.4: > utils-server:/home/igor# lspci -s 00:1f.4 -nnv > 00:1f.4 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 03) > Subsystem: Red Hat, Inc. QEMU Virtual Machine [1af4:1100] > Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 18 > Memory at e1080000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+ > Kernel driver in use: vfio-pci > > to achieve that, I've compiled the kernel with acs patch and defined it in the kernel cmdline: > BOOT_IMAGE=/boot/vmlinuz-lts root=UUID=44299ace-27c6-4047-8cbd-bbffcc0a65f0 ro modules=sd-mod,usb-storage,ext4 quiet rootfstype=ext4 iommu=pt intel_iommu=on pcie_acs_override=id:8086:293e,8086:10c9 > > I see in dmesg, acs is enabled, see: > # dmesg | grep "ACS overrides" > [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA > > the xml file can be found at https://bpa.st/FARQ. > when I try to start the vm, I get this: > # virsh start streamer > error: Failed to start domain 'streamer' > error: internal error: QEMU unexpectedly closed the monitor (vm='streamer'): 2024-06-21T08:39:17.959476Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:1f.4","id":"hostdev0","bus":"pcie.0","addr":"0x1f.0x5"}: vfio 0000:00:1f.4: group 18 is not viable > Please ensure all devices within the iommu_group are bound to their vfio bus driver > > any ideas what am I missing? is it possible this cannot work within a vm? > > Thanks, > > Dagg > answering to myself, I needed to unbind all the devices in the same iommu group. hope it helps anyone else
