On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
Hello!
I've got a few questions to a problem, which already was analyzed here
sometime ago:
http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
My situation is a bit different. I do have two PCI cards (a Linksys wlan
card and an intel e100 card). Each of these cards should be managed in
an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
The problem is: both cards are behind a PCI-PCI bridge:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
-[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
+-00.2 ATI Technologies Inc Device 5a23
+-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
| \-00.1 ATI Technologies Inc Device aa90
+-04.0-[02]----00.0 Device 1b6f:7023
+-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
+-0a.0-[04]----00.0 Device 1b6f:7023
+-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
+-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-14.0 ATI Technologies Inc SBx00 SMBus Controller
+-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
+-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
+-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
+-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
| \-07.0 RaLink RT2800 802.11n PCI
You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
different VMs because PCI devices dont provide a B:D.F in their transaction headers;
In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
a transaction that is sourced by 05:06.0 or 05:07.0 .
So, the devices cannot be isolated from each other's (mem) mapping domains in the
IOMMU, thus, libvirt prevents this (security) violation from being enabled.
+-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
+-15.0-[06]--
+-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-18.0 Advanced Micro Devices [AMD] Device 1600
+-18.1 Advanced Micro Devices [AMD] Device 1601
+-18.2 Advanced Micro Devices [AMD] Device 1602
+-18.3 Advanced Micro Devices [AMD] Device 1603
+-18.4 Advanced Micro Devices [AMD] Device 1604
\-18.5 Advanced Micro Devices [AMD] Device 1605
That's what can be seen during boot:
[ 0.621901] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
[ 0.621906] AMD-Vi: mmio-addr: 00000000fec30000
[ 0.622091] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00
[ 0.622095] AMD-Vi: DEV_RANGE_END devid: 00:00.2
[ 0.622097] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00
[ 0.622100] AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00
[ 0.622102] AMD-Vi: DEV_RANGE_END devid: 01:00.1
[ 0.622105] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00
[ 0.622107] AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00
[ 0.622109] AMD-Vi: DEV_SELECT devid: 00:09.0 flags: 00
[ 0.622112] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00
[ 0.622114] AMD-Vi: DEV_SELECT devid: 00:0a.0 flags: 00
[ 0.622117] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00
[ 0.622119] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00
[ 0.622122] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00
[ 0.622124] AMD-Vi: DEV_RANGE_END devid: 00:12.2
[ 0.622127] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00
[ 0.622129] AMD-Vi: DEV_RANGE_END devid: 00:13.2
[ 0.622132] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7
[ 0.622134] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00
[ 0.622137] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00
[ 0.622139] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00
[ 0.622141] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00
[ 0.622144] AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4
[ 0.622147] AMD-Vi: DEV_RANGE_END devid: 05:1f.7
[ 0.622154] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00
[ 0.622157] AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00
[ 0.622159] AMD-Vi: DEV_SELECT_RANGE_START devid: 06:00.0 flags: 00
[ 0.622162] AMD-Vi: DEV_RANGE_END devid: 06:1f.7
[ 0.622166] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00
[ 0.622169] AMD-Vi: DEV_RANGE_END devid: 00:16.2
[ 0.622176] pci 0000:00:00.2: can't derive routing for PCI INT A
[ 0.622179] pci 0000:00:00.2: PCI INT A: no GSI
[ 0.622379] pci 0000:00:00.2: irq 40 for MSI/MSI-X
[ 0.622453] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
[ 0.681866] AMD-Vi: Lazy IO/TLB flushing enabled
[ 0.681870] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.681873] Placing 64MB software IO TLB between ffff8800cbd9d000 - ffff8800cfd9d000
[ 0.681877] software IO TLB at phys 0xcbd9d000 - 0xcfd9d000
Unbind is done like this:
# PCI bridge
echo "1002 4384"> /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:14.4"> /sys/bus/pci/drivers/pci-stub/unbind
# e100
echo "8086 1229"> /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:06.0"> /sys/bus/pci/drivers/pci-stub/unbind
echo "0000:05:06.0"> /sys/bus/pci/drivers/pci-stub/bind
echo "8086 1229"> /sys/bus/pci/drivers/pci-stub/remove_id
# Linksys
echo "1814 0601"> /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:07.0"> /sys/bus/pci/drivers/pci-stub/unbind
echo "0000:05:07.0"> /sys/bus/pci/drivers/pci-stub/bind
echo "1814 0601"> /sys/bus/pci/drivers/pci-stub/remove_id
I can start a VM (and can see the devices in the VM) with both devices or just one PCI device (without the bridge). If the machine in this VM is halted, the host crashes as already described here:
http://markmail.org/message/dspovwvzp3wtdrf6
(The bridge can't be put to the VM because libvirt (I'm using libvirt) doesn't like it.)
Unfortunately, it wasn't possible to put the PCI devices to different VMs at the same time (Failed to assign device "hostdev0" : Device or resource busy) - but that's exactly what I need :-(.
Now my questions:
1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
Only if the two slots are behind different PCIe-PCI bridges .
2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
for assigned devices, so the above echo steps are unnecessary if you use libvirt.
Thank you very much for your help,
kind regards,
Andreas Hartmann
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
