Hi Alex, Yes. They are PCIe devices which expose the PCIe functionality: -bash-4.1# lspci -vv -s 04:00 …. Capabilities: [ac] Express (v2) Endpoint, MSI 00 -bash-4.1# lspci -vv -s 03:00 …. Capabilities: [80] Express (v2) Endpoint, MSI 00 Is there any dependency issue here ? Does KVM expect the downstream ports of the PCIe switch also expected to be passed through ? Thanks, Ganesh On Mar 20, 2013, at 7:41 PM, Alex Williamson wrote: > On Tue, 2013-03-19 at 17:09 -0700, Ganesh Narayanaswamy wrote: >> Hi Alex, >> >> Thanks for your reply. The pci devices in question are proprietary FPGAs. Here is the lspci -tv output: >> >> -bash-4.1# lspci -tv >> -[0000:00]-+-00.0 Intel Corporation Sandy Bridge DRAM Controller >> +-01.0-[01-04]----00.0-[02-04]--+-01.0-[03]----00.0 Broadcom Corporation Device b850 >> | \-02.0-[04]----00.0 Broadcom Corporation Device b850 >> +-01.1-[05]-- >> +-06.0-[06]--+-00.0 Intel Corporation Device 0434 >> | +-00.1 Intel Corporation Device 0438 >> | +-00.2 Intel Corporation Device 0438 >> | +-00.3 Intel Corporation Device 0436 >> | \-00.4 Intel Corporation Device 0436 >> +-1d.0 Intel Corporation Device 2334 >> +-1f.0 Intel Corporation Device 2310 >> +-1f.2 Intel Corporation Device 2323 >> +-1f.3 Intel Corporation Device 2330 >> +-1f.4 Intel Corporation Device 2331 >> +-1f.6 Intel Corporation Device 2332 >> \-1f.7 Intel Corporation Device 2360 >> >> My qemu command line is as follows: >> >> qemu-system-x86_64 -M q35 --enable-kvm -m 2048 -nographic -vga std >> -usb -drive file=<IMG file>,if=none,id=drive-sata-disk0,format=raw >> -device ahci,id=ahci -device >> ide-drive,bus=ahci.0,drive=drive-sata-disk0,id=sata-disk0,bootindex=1 >> -device pci-assign,host=04:00.0 -device pci-assign,host=03:00.0 >> >> >> The PCIe bridge is a PLX 8613 device: >> >> 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8613 12-lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) >> 02:01.0 PCI bridge: PLX Technology, Inc. PEX 8613 12-lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) >> 02:02.0 PCI bridge: PLX Technology, Inc. PEX 8613 12-lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba) >> >> As shown by the lspci -tv output, each of the PCI device being passed >> through is connected to one of the downstream ports of the PLX PCI >> bridge. > > Are your FPGAs actually PCIe devices (they must be because they connect > to a PCIe switch) that do not expose a PCIe capability? For example, > lspci -v: > > Capabilities: [e0] Express Endpoint, MSI 00 > > If so, they're in violation of the PCI Express specification and likely > the cause of this problem. Thanks, > > Alex > >> On Mar 19, 2013, at 3:28 PM, Alex Williamson wrote: >> >>> On Tue, 2013-03-19 at 13:30 -0700, Ganesh Narayanaswamy wrote: >>>> Hi, >>>> >>>> I am running qemu with kvm and VT-d enabled and a couple of PCI >>>> devices assigned to the guest VM. Both host and guest are running >>>> linux 2.6 kernel. >>>> >>>> The passthrough works fine, but when I exit the VM, the host kernel >>>> crashes with the following backtrace: >>>> >>>> <4>[ 5569.836893] Process qemu-system-x86 (pid: 2925, threadinfo ffff8801f5f40000, task ffff88024fa28720) >>>> <0>[ 5569.944946] Stack: >>>> <4>[ 5569.968845] ffff8801f5f41aa8 ffffffff811a45fb ffff88024f04b680 ffff88024f049980 >>>> <4>[ 5570.057156] ffff88024f04b680 ffff88024f049988 ffff8801f5f41b08 ffffffff811a6371 >>>> <4>[ 5570.145470] ffff8801f5f41ad8 ffffffff81391045 0000000000000246 ffff88024f049990 >>>> <0>[ 5570.233785] Call Trace: >>>> <4>[ 5570.262880] [<ffffffff811a45fb>] iommu_detach_dependent_devices+0x25/0x91 >>>> <4>[ 5570.344958] [<ffffffff811a6371>] vm_domain_exit+0xf8/0x28b >>>> <4>[ 5570.411457] [<ffffffff81391045>] ? sub_preempt_count+0x92/0xa6 >>>> <4>[ 5570.482106] [<ffffffff811a651a>] intel_iommu_domain_destroy+0x16/0x18 >>>> <4>[ 5570.560030] [<ffffffff811fb5ea>] iommu_domain_free+0x16/0x22 >>>> <4>[ 5570.628611] [<ffffffffa0006261>] kvm_iommu_unmap_guest+0x22/0x28 [kvm] >>>> <4>[ 5570.707570] [<ffffffffa0009b7b>] kvm_arch_destroy_vm+0x19/0x12a [kvm] >>>> <4>[ 5570.785492] [<ffffffffa0002614>] kvm_put_kvm+0xe6/0x129 [kvm] >>>> <4>[ 5570.855102] [<ffffffffa0002eb3>] kvm_vcpu_release+0x13/0x17 [kvm] >>>> <4>[ 5570.928867] [<ffffffff8109cdfc>] fput+0x117/0x1be >>>> <4>[ 5570.986013] [<ffffffff8109a147>] filp_close+0x63/0x6d >>>> <4>[ 5571.047314] [<ffffffff810342dd>] put_files_struct+0x6f/0xda >>>> <4>[ 5571.114845] [<ffffffff8103438e>] exit_files+0x46/0x4e >>>> <4>[ 5571.176145] [<ffffffff81035b3d>] do_exit+0x1fc/0x681 >>>> <4>[ 5571.236416] [<ffffffffa000dedc>] ? kvm_arch_vcpu_ioctl_run+0xc2d/0xc55 [kvm] >>>> <4>[ 5571.321605] [<ffffffff8138cc41>] ? __mutex_lock_slowpath+0x26c/0x294 >>>> <4>[ 5571.398490] [<ffffffff81036034>] do_group_exit+0x72/0x9a >>>> <4>[ 5571.462907] [<ffffffff8103fec9>] get_signal_to_deliver+0x331/0x350 >>>> <4>[ 5571.537719] [<ffffffff81001f0f>] do_signal+0x6d/0x69a >>>> <4>[ 5571.599013] [<ffffffff811da1fc>] ? put_ldisc+0x92/0x97 >>>> <4>[ 5571.661353] [<ffffffff810a95ea>] ? do_vfs_ioctl+0x527/0x576 >>>> <4>[ 5571.728887] [<ffffffff81002563>] do_notify_resume+0x27/0x51 >>>> <4>[ 5571.796419] [<ffffffff810a968c>] ? sys_ioctl+0x53/0x65 >>>> <4>[ 5571.858758] [<ffffffff81002b9b>] int_signal+0x12/0x17 >>>> <0>[ 5571.920058] Code: 48 85 d2 0f 95 c0 c9 c3 55 80 7f 4a 00 48 89 f8 48 89 e5 75 46 31 d2 48 8b 40 10 48 83 78 10 00 75 05 48 89 d0 eb 36 48 8b 40 38 <80> 78 4a 00 48 89 c2 74 e3 80 78 4b 07 74 23 80 3d 86 b5 5a 00 >>>> <1>[ 5572.145516] RIP [<ffffffff81197f8c>] pci_find_upstream_pcie_bridge+0x23/0x57 >>>> <4>[ 5572.230712] RSP <ffff8801f5f41a78> >>>> >>>> The two PCI devices in question are behind a PCIe bridge which is >>>> connected to the rootport. The crash seems to be happening when >>>> cleaning up the PCIe tree of the passed-through PCI devices. I tried >>>> passing through the downstream ports of the bridge as well, but that >>>> is not supported by qemu. >>>> >>>> Am I doing something wrong/unexpected here ? Any help in understanding >>>> this issue will help me fix the issue properly. >>> >>> Please provide 'sudo lspci -vvv' from the host and the qemu commandline >>> you're using. Is the bridge by chance an asmedia device? Thanks, >>> >>> Alex >>> >> > > > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
