I enabled iommu in the grub-config: grep iommu /boot/grub2/grub.cfg linux16 /vmlinuz-3.10.0-123.8.1.el7.x86_64 root=UUID=9cfb58ce-b401-47d1-8dc0-babbd1f21ec3 ro rootflags=subvol=root crashkernel=auto vconsole.keymap=de vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on Here is the relevant output of /var/log/messages during boot: Oct 7 09:18:01 artemis2 kernel: dmar: Host address width 39 Oct 7 09:18:01 artemis2 kernel: dmar: DRHD base: 0x000000fed90000 flags: 0x1 Oct 7 09:18:01 artemis2 kernel: dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da Oct 7 09:18:01 artemis2 kernel: dmar: RMRR base: 0x000000ceefa000 end: 0x000000cef06fff Oct 7 09:18:01 artemis2 kernel: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0 Later on I get the following error: Oct 7 09:18:01 artemis2 kernel: DMAR: No ATSR found Oct 7 09:18:01 artemis2 kernel: IOMMU 0 0xfed90000: using Queued invalidation Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting RMRR: Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1d.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1a.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:14.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Prepare 0-16MiB unity mapping for LPC Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] Oct 7 09:18:01 artemis2 kernel: PCI-DMA: Intel(R) Virtualization Technology for Directed I/O Oct 7 09:18:01 artemis2 kernel: ------------[ cut here ]------------ Oct 7 09:18:01 artemis2 kernel: WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x87/0x90() Oct 7 09:18:01 artemis2 kernel: Modules linked in: Oct 7 09:18:01 artemis2 kernel: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.8.1.el7.x86_64 #1 Oct 7 09:18:01 artemis2 kernel: Hardware name: Hewlett-Packard HP Z230 Tower Workstation/1905, BIOS L51 v01.13 10/18/2013 Oct 7 09:18:01 artemis2 kernel: 0000000000000000 00000000eccfd72f ffff8802220e3cf0 ffffffff815e237b Oct 7 09:18:01 artemis2 kernel: ffff8802220e3d28 ffffffff8105dee1 ffff880221f95000 ffff880221f9f000 Oct 7 09:18:01 artemis2 kernel: ffff880221f9f098 0000000000000000 0000000000000000 ffff8802220e3d38 Oct 7 09:18:01 artemis2 kernel: Call Trace: Oct 7 09:18:01 artemis2 kernel: [<ffffffff815e237b>] dump_stack+0x19/0x1b Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80 Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20 Oct 7 09:18:01 artemis2 kernel: [<ffffffff812ec607>] pci_find_upstream_pcie_bridge+0x87/0x90 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814aa04d>] intel_iommu_add_device+0x4d/0x230 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10f0>] ? bus_set_iommu+0x50/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a111a>] add_iommu_group+0x2a/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff813b4443>] bus_for_each_dev+0x73/0xc0 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10e8>] bus_set_iommu+0x48/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a5837d>] intel_iommu_init+0x3ea/0x413 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11c93>] ? memblock_find_dma_reserve+0x147/0x147 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11ca5>] pci_iommu_init+0x12/0x3c Oct 7 09:18:01 artemis2 kernel: [<ffffffff810020e2>] do_one_initcall+0xe2/0x190 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a09153>] kernel_init_freeable+0x18b/0x22a Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a0892b>] ? do_early_param+0x88/0x88 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c2e>] kernel_init+0xe/0x180 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80 Oct 7 09:18:01 artemis2 kernel: ---[ end trace 56f25a222bb09abf ]--- Best regards Michael -----Ursprüngliche Nachricht----- Von: libvirt-users-bounces@xxxxxxxxxx [mailto:libvirt-users-bounces@xxxxxxxxxx] Im Auftrag von Pierre Schweitzer Gesendet: Montag, 13. Oktober 2014 13:19 An: libvirt-users@xxxxxxxxxx Betreff: Re: passthrough of PCI-device And what about IO MMU ? On 10/13/2014 12:02 PM, Weis, Michael (DWIE) wrote: > Hi Pierre, > > thanks for your reply. > > I am using kernel 3.10.0-123.8.1.el7.x86_64. > The kernel modul used after nodedev-detach is vfio-pci > > This is the output of lspci -vv after I did a virsh nodedev-detatch > pci_0000_02_00_0 > > 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e > Subsystem: PLX Technology, Inc. Device 235e > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Interrupt: pin A routed to IRQ 16 > Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256] > Region 1: I/O ports at e000 [disabled] [size=256] > Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M] > Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M] > Expansion ROM at f0800000 [disabled] [size=64K] > Capabilities: [40] Power Management version 1 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: [48] CompactPCI hot-swap <?> > Capabilities: [4c] Vital Product Data > Unknown large resource type 35, will not decode more. > Kernel driver in use: vfio-pci > > > Best regards > > Michael > > > > > -----Ursprüngliche Nachricht----- > Von: libvirt-users-bounces@xxxxxxxxxx > [mailto:libvirt-users-bounces@xxxxxxxxxx] Im Auftrag von Pierre > Schweitzer > Gesendet: Montag, 13. Oktober 2014 10:04 > An: libvirt-users@xxxxxxxxxx > Betreff: Re: passthrough of PCI-device > > Dear Michael, > > Did you activate the Intel IO MMU (or its equivalent for AMD)? > Also, did you load the pci_stub module for Linux? It is mandatory (it will replace current driver for your passed through hardware). > > Cheers, > Pierre > > On 10/13/2014 07:54 AM, Weis, Michael (DWIE) wrote: >> Good morning, >> >> there is a typo in my description; >> the line >> <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/> >> should be >> <address domain='0x0' bus='0x2' slot='0x00' function='0x0'/> >> That was correct in my xml-file. >> >> Isn't there anybody how can help me with that? >> >> >> Regards >> >> Michael Weis >> >> >> >> Von: Weis, Michael (DWIE) >> Gesendet: Dienstag, 7. Oktober 2014 11:45 >> An: 'libvirt-users@xxxxxxxxxx' >> Betreff: passthrough of PCI-device >> >> Hello, >> >> I try to passthrough a PCI-card to a VM named testvm I want to do >> that with an xml-file named hga.xml including the following content: >> >> <hostdev mode='subsystem' type='pci' managed='yes'> >> <source> >> <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/> >> </source> >> </hostdev> >> >> When I execute >> virsh attach-device testvm hga.xml >> I get the following error: >> Fehler: Fehler beim Anhängen des Geräts von hga.xml >> (error: error while attaching device from hga.xml) >> Fehler: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed. >> (error: internal error: cannot execute 'device_add': >> >> In the syslog of the host I get the following errors: >> Oct 7 11:32:34 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed. >> Oct 7 11:32:53 artemis2 kernel: ffff880220fadd10 ffffffff810f924a >> ffff880220fadd28 0000000000000246 Oct 7 11:32:53 artemis2 kernel: >> ffff8800bd07a400 000000007265756d ffffffffa0866ee0 0000000000000000 Oct 7 11:32:53 artemis2 kernel: Call Trace: >> Oct 7 11:32:53 artemis2 kernel: [<ffffffff815e237b>] >> dump_stack+0x19/0x1b Oct 7 11:32:53 artemis2 kernel: >> [<ffffffff810f924a>] __setup_irq+0x54a/0x570 Oct 7 11:32:53 artemis2 >> kernel: [<ffffffffa0866ee0>] ? virqfd_wakeup+0xc0/0xc0 [vfio_pci] Oct >> 7 11:32:53 artemis2 kernel: [<ffffffff810f93cc>] >> request_threaded_irq+0xcc/0x170 Oct 7 11:32:53 artemis2 kernel: >> [<ffffffffa0866b8a>] vfio_intx_set_signal+0x10a/0x1f0 [vfio_pci] Oct >> 7 11:32:53 artemis2 kernel: [<ffffffffa08670e8>] >> vfio_pci_set_intx_trigger+0xc8/0x1b0 [vfio_pci] Oct 7 11:32:53 >> artemis2 kernel: [<ffffffffa086799b>] >> vfio_pci_set_irqs_ioctl+0x7b/0x100 [vfio_pci] Oct 7 11:32:53 >> artemis2 >> kernel: [<ffffffff8115d352>] ? memdup_user+0x42/0x70 Oct 7 11:32:53 >> artemis2 kernel: [<ffffffffa0865e74>] vfio_pci_ioctl+0x314/0xa00 >> [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff81194b6a>] ? >> kmem_cache_alloc+0x1ba/0x1d0 Oct 7 11:32:53 artemis2 kernel: >> [<ffffffff811b156c>] ? get_empty_filp+0x5c/0x190 Oct 7 11:32:53 >> artemis2 kernel: [<ffffffff811f1891>] ? fsnotify+0x251/0x350 Oct 7 >> 11:32:53 artemis2 kernel: [<ffffffff81251806>] ? >> security_file_alloc+0x16/0x20 Oct 7 11:32:53 artemis2 kernel: >> [<ffffffff811b15a0>] ? get_empty_filp+0x90/0x190 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa084f163>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2af5>] do_vfs_ioctl+0x2e5/0x4c0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811af780>] ? vfs_write+0x160/0x1e0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2d71>] SyS_ioctl+0xa1/0xc0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff815f2a59>] system_call_fastpath+0x16/0x1b Oct 7 11:32:53 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed. >> >> My version-information: >> OS: CentOS Linux release 7.0.1406 (Core) >> Kernel: Linux artemis2 3.10.0-123.8.1.el7.x86_64 >> libvirt-daemon-driver-secret-1.2.8-1.el7.centos.x86_64 >> libvirt-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-kvm-1.2.8-1.el7.centos.x86_64 >> libvirt-client-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-qemu-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-config-network-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-interface-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-nodedev-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-storage-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-config-nwfilter-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-network-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-lxc-1.2.8-1.el7.centos.x86_64 >> libvirt-daemon-driver-nwfilter-1.2.8-1.el7.centos.x86_64 >> >> The PCI-Card I want to passthrough: >> 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e >> Subsystem: PLX Technology, Inc. Device 235e >> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- >> Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- >> Interrupt: pin A routed to IRQ 16 >> Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256] >> Region 1: I/O ports at e000 [disabled] [size=256] >> Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M] >> Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M] >> Expansion ROM at f0800000 [disabled] [size=64K] >> Capabilities: [40] Power Management version 1 >> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) >> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- >> Capabilities: [48] CompactPCI hot-swap <?> >> Capabilities: [4c] Vital Product Data >> Unknown large resource type 35, will not decode more. >> Kernel driver in use: vfio-pci >> >> Please help. >> >> >> Best regards >> >> Michael Weis >> >> >> >> PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail. >> >> >> >> _______________________________________________ >> libvirt-users mailing list >> libvirt-users@xxxxxxxxxx >> https://www.redhat.com/mailman/listinfo/libvirt-users >> > > > -- > Pierre Schweitzer <pierre@xxxxxxxxxxx> System & Network Administrator > Senior Kernel Developer ReactOS Deutschland e.V. > > > ______________________________________________________________________ > CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed. > > PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail. > > _______________________________________________ > libvirt-users mailing list > libvirt-users@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/libvirt-users > -- Pierre Schweitzer <pierre@xxxxxxxxxxx> System & Network Administrator Senior Kernel Developer ReactOS Deutschland e.V. ______________________________________________________________________ CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed. PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail. _______________________________________________ libvirt-users mailing list libvirt-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvirt-users
