Hello,
I come to this problem when using PCI passthrough (AMD IOMMU), I need
to passthrough a single PCI device which is behind PCIe-to-PCI bridge.
PCI tree in this case;
# lspci -t -v
-[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge
(external gfx0 port A)
+-00.2 ATI Technologies Inc RD990 I/O Memory Management Unit (IOMMU)
+-09.0-[01]----00.0 Intel Corporation 82574L Gigabit
Network Connection
+-0a.0-[02]----00.0 Intel Corporation 82574L Gigabit
Network Connection
+-11.0 ATI Technologies Inc SB7x0/SB8x0/SB9x0 SATA
Controller [AHCI mode]
+-12.0 ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
+-12.1 ATI Technologies Inc SB7x0 USB OHCI1 Controller
+-12.2 ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB EHCI Controller
+-13.0 ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
+-13.1 ATI Technologies Inc SB7x0 USB OHCI1 Controller
+-13.2 ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB EHCI Controller
+-14.0 ATI Technologies Inc SBx00 SMBus Controller
+-14.3 ATI Technologies Inc SB7x0/SB8x0/SB9x0 LPC host controller
+-14.4-[03]--+-03.0 Creative Labs SB Audigy
| +-03.1 Creative Labs SB Audigy Game Port
| +-03.2 Creative Labs SB Audigy FireWire Port
| \-04.0 Matrox Graphics, Inc. MGA G200eW WPCM450
+-14.5 ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
+-18.0 Advanced Micro Devices [AMD] Family 10h Processor
HyperTransport Configuration
+-18.1 Advanced Micro Devices [AMD] Family 10h Processor Address Map
+-18.2 Advanced Micro Devices [AMD] Family 10h Processor
DRAM Controller
+-18.3 Advanced Micro Devices [AMD] Family 10h Processor
Miscellaneous Control
\-18.4 Advanced Micro Devices [AMD] Family 10h Processor
Link Control
I am passing-through only "03.0 Creative Labs SB Audigy" device which
is just enough for the case, but to be able to do so i need to unbind
all devices sharing the same bridge, as well as the bridge device
itself before starting KVM guest;
# echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
# echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind
The actual passthrough works very good at guest side, but there is a
problem when i shut the guest down;
Jan 14 03:29:33 H8SCM kernel: [ 1413.439703] ------------[ cut here
]------------
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] kernel BUG at
/build/buildd/linux-3.2.0/drivers/iommu/amd_iommu.c:1740!
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] invalid opcode: 0000 [#1] SMP
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] CPU 0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Modules linked in:
pci_stub ip6table_filter ip6_tables ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp
iptable_filter ip_tables x_tables kvm_amd kvm vesafb w83795 jc42
snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul
bridge stp snd_emu10k1 snd_ac97_codec ac97_bus snd_pcm snd_page_alloc
snd_util_mem snd_hwdep snd_seq_midi snd_rawmidi snd_seq_midi_event
psmouse snd_seq joydev snd_timer snd_seq_device usbhid serio_raw snd
emu10k1_gp amd64_edac_mod gameport soundcore edac_core k10temp
edac_mce_amd hid sp5100_tco i2c_piix4 ipmi_si ipmi_msghandler lp
parport firewire_ohci firewire_core crc_itu_t e1000e
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Pid: 1550, comm: kvm Not
tainted 3.2.0-7-generic #13-Ubuntu Supermicro H8SCM/H8SCM
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RIP:
0010:[<ffffffff81512896>] [<ffffffff81512896>]
__detach_device+0xc6/0xd0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RSP:
0018:ffff88020ea57ac8 EFLAGS: 00010046
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RAX: dead000000100100
RBX: ffff8802103dd640 RCX: 00000000fffffff0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RDX: ffff88020ea57a68
RSI: 0000000000000082 RDI: ffff8802103dd640
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RBP: ffff88020ea57ae8
R08: 0000000000002000 R09: dead000000100100
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] R10: dead000000200200
R11: 0000000000000000 R12: 0000000000000000
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] R13: dead000000100100
R14: ffff8802103dd640 R15: ffff88020d634410
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] FS:
00007fb8d2b9b700(0000) GS:ffff88021fc00000(0000)
knlGS:0000000000000000
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] CS: 0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] CR2: 0000000000d03000
CR3: 000000020fe51000 CR4: 00000000000006f0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Process kvm (pid: 1550,
threadinfo ffff88020ea56000, task ffff88020d5d0000)
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Stack:
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] ffff88020d634400
ffff8802110ffbe0 dead000000100100 ffff8802103dd640
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] ffff88020ea57b38
ffffffff8151371e ffff88020f170038 0000000000000206
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] ffff88020fa88000
ffff8802110ffbe0 ffff880211edad80 0000000000000003
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Call Trace:
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8151371e>]
amd_iommu_domain_destroy+0x9e/0xd0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff815110cf>]
iommu_domain_free+0x1f/0x30
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffffa0246117>]
kvm_iommu_unmap_guest+0x27/0x30 [kvm]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffffa02586e8>]
kvm_arch_destroy_vm+0x18/0x150 [kvm]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffffa02408cd>]
kvm_destroy_vm+0xed/0x140 [kvm]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffffa0240945>]
kvm_put_kvm+0x25/0x30 [kvm]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffffa0240c88>]
kvm_vcpu_release+0x18/0x20 [kvm]
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8117776e>]
__fput+0xbe/0x210
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff811778e5>]
fput+0x25/0x30
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81174366>]
filp_close+0x66/0x90
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81067aaa>]
put_files_struct.part.10+0x7a/0xe0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff810694f8>]
put_files_struct+0x18/0x20
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff810695c4>]
exit_files+0x54/0x70
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81069aa5>]
do_exit+0x195/0x420
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8107820a>] ?
__dequeue_signal+0x6a/0xb0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81069ed4>]
do_group_exit+0x44/0xa0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8107ad8c>]
get_signal_to_deliver+0x21c/0x420
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81013815>]
do_signal+0x45/0x130
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8109e88c>] ?
do_futex+0xbc/0x1d0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8109eaaa>] ?
sys_futex+0x10a/0x1a0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81176400>] ?
vfs_write+0x110/0x180
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff81013ac5>]
do_notify_resume+0x65/0x80
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] [<ffffffff8165a890>]
int_signal+0x12/0x17
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Code: 0f 1f 44 00 00 e8
9b fe ff ff eb aa 66 0f 1f 84 00 00 00 00 00 48 8b 35 01 9f a0 00 49
39 f4 74 be 48 89 df e8 2c fd ff ff eb b4 <0f> 0b 0f 1f 84 00 00 00 00
00 55 48 89 e5 48 83 ec 20 48 89 5d
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RIP
[<ffffffff81512896>] __detach_device+0xc6/0xd0
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] RSP <ffff88020ea57ac8>
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] ---[ end trace
a18be2ece9e040e5 ]---
Jan 14 03:29:33 H8SCM kernel: [ 1413.440001] Fixing recursive fault
but reboot is needed!
Well, something similar was reported almost a year ago, ("PCI
Passthrough, error: The driver 'pci-stub' is occupying your device
0000:08:06.2", http://www.spinics.net/lists/kvm/msg50687.html) but the
problems seems to manifest itself in detach_device() in my case
instead of amd_iommu_domain_destroy() as was reported. Please share if
there are any news about possible fix or workaround. I am willing to
invest some time in trying to investigate this if someone would kindly
point me to a proper place in amd_iommu.c.
Cheers,
Samir
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
