PCI Passthrough, amd_iommu detach_device error when unbind PCIe-to-PCI bridge

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I come to this problem when using PCI passthrough (AMD IOMMU), I need
to passthrough a single PCI device which is behind PCIe-to-PCI bridge.
PCI tree in this case;

# lspci -t -v
-[0000:00]-+-00.0  ATI Technologies Inc RD890 PCI to PCI bridge
(external gfx0 port A)
           +-00.2  ATI Technologies Inc RD990 I/O Memory Management Unit (IOMMU)
           +-09.0-[01]----00.0  Intel Corporation 82574L Gigabit
Network Connection
           +-0a.0-[02]----00.0  Intel Corporation 82574L Gigabit
Network Connection
           +-11.0  ATI Technologies Inc SB7x0/SB8x0/SB9x0 SATA
Controller [AHCI mode]
           +-12.0  ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
           +-12.1  ATI Technologies Inc SB7x0 USB OHCI1 Controller
           +-12.2  ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB EHCI Controller
           +-13.0  ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
           +-13.1  ATI Technologies Inc SB7x0 USB OHCI1 Controller
           +-13.2  ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB EHCI Controller
           +-14.0  ATI Technologies Inc SBx00 SMBus Controller
           +-14.3  ATI Technologies Inc SB7x0/SB8x0/SB9x0 LPC host controller
           +-14.4-[03]--+-03.0  Creative Labs SB Audigy
           |            +-03.1  Creative Labs SB Audigy Game Port
           |            +-03.2  Creative Labs SB Audigy FireWire Port
           |            \-04.0  Matrox Graphics, Inc. MGA G200eW WPCM450
           +-14.5  ATI Technologies Inc SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
           +-18.0  Advanced Micro Devices [AMD] Family 10h Processor
HyperTransport Configuration
           +-18.1  Advanced Micro Devices [AMD] Family 10h Processor Address Map
           +-18.2  Advanced Micro Devices [AMD] Family 10h Processor
DRAM Controller
           +-18.3  Advanced Micro Devices [AMD] Family 10h Processor
Miscellaneous Control
           \-18.4  Advanced Micro Devices [AMD] Family 10h Processor
Link Control

I am passing-through only "03.0  Creative Labs SB Audigy" device which
is just enough for the case, but to be able to do so i need to unbind
all devices sharing the same bridge, as well as the bridge device
itself before starting KVM guest;

# echo 1002 4384 > /sys/bus/pci/drivers/pci-stub/new_id
# echo 0000:00:14.4 > /sys/bus/pci/drivers/pci-stub/unbind

The actual passthrough works very good at guest side, but there is a
problem when i shut the guest down;

Jan  14 03:29:33 H8SCM kernel: [ 1413.439703] ------------[ cut here
]------------
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] kernel BUG at
/build/buildd/linux-3.2.0/drivers/iommu/amd_iommu.c:1740!
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] invalid opcode: 0000 [#1] SMP
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] CPU 0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Modules linked in:
pci_stub ip6table_filter ip6_tables ebtable_nat ebtables
ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp
iptable_filter ip_tables x_tables kvm_amd kvm vesafb w83795 jc42
snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul
bridge stp snd_emu10k1 snd_ac97_codec ac97_bus snd_pcm snd_page_alloc
snd_util_mem snd_hwdep snd_seq_midi snd_rawmidi snd_seq_midi_event
psmouse snd_seq joydev snd_timer snd_seq_device usbhid serio_raw snd
emu10k1_gp amd64_edac_mod gameport soundcore edac_core k10temp
edac_mce_amd hid sp5100_tco i2c_piix4 ipmi_si ipmi_msghandler lp
parport firewire_ohci firewire_core crc_itu_t e1000e
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Pid: 1550, comm: kvm Not
tainted 3.2.0-7-generic #13-Ubuntu Supermicro H8SCM/H8SCM
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RIP:
0010:[<ffffffff81512896>]  [<ffffffff81512896>]
__detach_device+0xc6/0xd0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RSP:
0018:ffff88020ea57ac8  EFLAGS: 00010046
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RAX: dead000000100100
RBX: ffff8802103dd640 RCX: 00000000fffffff0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RDX: ffff88020ea57a68
RSI: 0000000000000082 RDI: ffff8802103dd640
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RBP: ffff88020ea57ae8
R08: 0000000000002000 R09: dead000000100100
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] R10: dead000000200200
R11: 0000000000000000 R12: 0000000000000000
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] R13: dead000000100100
R14: ffff8802103dd640 R15: ffff88020d634410
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] FS:
00007fb8d2b9b700(0000) GS:ffff88021fc00000(0000)
knlGS:0000000000000000
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] CS:  0010 DS: 0000 ES:
0000 CR0: 000000008005003b
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] CR2: 0000000000d03000
CR3: 000000020fe51000 CR4: 00000000000006f0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] DR0: 0000000000000000
DR1: 0000000000000000 DR2: 0000000000000000
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] DR3: 0000000000000000
DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Process kvm (pid: 1550,
threadinfo ffff88020ea56000, task ffff88020d5d0000)
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Stack:
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  ffff88020d634400
ffff8802110ffbe0 dead000000100100 ffff8802103dd640
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  ffff88020ea57b38
ffffffff8151371e ffff88020f170038 0000000000000206
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  ffff88020fa88000
ffff8802110ffbe0 ffff880211edad80 0000000000000003
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Call Trace:
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8151371e>]
amd_iommu_domain_destroy+0x9e/0xd0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff815110cf>]
iommu_domain_free+0x1f/0x30
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffffa0246117>]
kvm_iommu_unmap_guest+0x27/0x30 [kvm]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffffa02586e8>]
kvm_arch_destroy_vm+0x18/0x150 [kvm]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffffa02408cd>]
kvm_destroy_vm+0xed/0x140 [kvm]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffffa0240945>]
kvm_put_kvm+0x25/0x30 [kvm]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffffa0240c88>]
kvm_vcpu_release+0x18/0x20 [kvm]
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8117776e>]
__fput+0xbe/0x210
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff811778e5>]
fput+0x25/0x30
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81174366>]
filp_close+0x66/0x90
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81067aaa>]
put_files_struct.part.10+0x7a/0xe0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff810694f8>]
put_files_struct+0x18/0x20
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff810695c4>]
exit_files+0x54/0x70
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81069aa5>]
do_exit+0x195/0x420
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8107820a>] ?
__dequeue_signal+0x6a/0xb0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81069ed4>]
do_group_exit+0x44/0xa0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8107ad8c>]
get_signal_to_deliver+0x21c/0x420
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81013815>]
do_signal+0x45/0x130
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8109e88c>] ?
do_futex+0xbc/0x1d0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8109eaaa>] ?
sys_futex+0x10a/0x1a0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81176400>] ?
vfs_write+0x110/0x180
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff81013ac5>]
do_notify_resume+0x65/0x80
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  [<ffffffff8165a890>]
int_signal+0x12/0x17
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Code: 0f 1f 44 00 00 e8
9b fe ff ff eb aa 66 0f 1f 84 00 00 00 00 00 48 8b 35 01 9f a0 00 49
39 f4 74 be 48 89 df e8 2c fd ff ff eb b4 <0f> 0b 0f 1f 84 00 00 00 00
00 55 48 89 e5 48 83 ec 20 48 89 5d
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] RIP
[<ffffffff81512896>] __detach_device+0xc6/0xd0
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001]  RSP <ffff88020ea57ac8>
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] ---[ end trace
a18be2ece9e040e5 ]---
Jan  14 03:29:33 H8SCM kernel: [ 1413.440001] Fixing recursive fault
but reboot is needed!

Well, something similar was reported almost a year ago, ("PCI
Passthrough, error: The driver 'pci-stub' is occupying your device
0000:08:06.2", http://www.spinics.net/lists/kvm/msg50687.html) but the
problems seems to manifest itself in detach_device() in my case
instead of amd_iommu_domain_destroy() as was reported. Please share if
there are any news about possible fix or workaround. I am willing to
invest some time in trying to investigate this if someone would kindly
point me to a proper place in amd_iommu.c.

Cheers,
Samir
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux