hi laine,
thx for the fast answer.
i tried VFIO as it seems supported and got at least one step further.
starting the guest with one of the DVB cards is possible now, even
when the other card is in the hosts pci config (not removed soft or
hard).
adding the second card/device thru ...
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <driver name='vfio'/>
> <source>
> <address domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <driver name='vfio'/>
> <source>
> <address domain='0x0000' bus='0x03' slot='0x01'
function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
> </hostdev>
will render the guest unbootable:
--><-- /var/log/libvirt/qemu/tvBackend.log
2014-03-12 16:48:48.018+0000: starting up
LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name tvBackend -S -machine
pc-i440fx-1.5,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp
2,sockets=2,cores=1,threads=1 -uuid
f647bb10-7f9a-f94c-33b9-3d99e8e753e0 -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/tvBackend.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/kvm/tvBackend.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:26:ec,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga std -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
W: kvm binary is deprecated, please use qemu-system-x86_64 instead
char device redirected to /dev/pts/0 (label charserial0)
qemu-system-x86_64: -device
vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6: Warning, device
0000:03:00.0 does not support reset
qemu-system-x86_64: -device
vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Warning, device
0000:03:01.0 does not support reset
qemu-system-x86_64: -device
vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: vfio: Error:
Failed to setup INTx fd: Device or resource busy
qemu-system-x86_64: -device
vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device
initialization failed.
qemu-system-x86_64: -device
vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device
'vfio-pci' could not be initialized
2014-03-12 16:48:48.639+0000: shutting down
--><-- /var/log/syslog
Mar 12 18:07:45 father kernel: [34317.290958] genirq: Flags mismatch
irq 16. 00000000 (vfio-intx(0000:03:01.0)) vs. 00000080
(ehci_hcd:usb1)
--><--
as i have only 2 pci slots on the board and no other board with
serious VT-d/IOMMU support i have no way to swap slots.
as the second card shared IRQ 16 with the ehci controller i
soft-removed the ehci controller for testing purposes and voila the
guest started with the 2 cards successfully inside, even mythtv works
... for now.
i hope to find a way to somehow rearrange the irqs as i sometimes need
USB on the machine :D
wbr,tja...
----- Ursprüngliche Mail -----
Von: "Laine Stump" <laine@xxxxxxxxx>
An: libvirt-users@xxxxxxxxxx
CC: "Thomas Jagoditsch" <tja@xxxxxxxxxxx>
Gesendet: Mittwoch, 12. März 2014 16:13:15
Betreff: Re: PCI Passthrough of 2 identical devices
On 03/12/2014 06:29 AM, Thomas Jagoditsch wrote:
hi,
i have a small trouble with pci-passthrough.
i have a working configuration passing a tv card into the guest, all
is fine and dandy.
as soon as i add my second identical card into the host i cant start
the guest anymore, whether i add the second card to the guest or not.
error message is identical in both cases.
message of virt-manager|virsh|libvirtd.log:
libvirtError: internal error: Unable to reset PCI device
0000:03:00.0: internal error: Active 0000:03:01.0 devices on bus with
0000:03:00.0, not doing bus reset
the host is a plain kvm server, no services or apps whatsoever
accessing the two devices.
the other guests (firewall and fileserver) do not use the cards
either.
The problem is that for kvm device assignment to work properly, the
device needs to be reset by libvirt after detaching it from the host
driver and before passing it to kvm, and these devices you are trying
to
reset support neither "function level reset" nor "power management
reset", so libvirt must fallback to resetting the entire bus where the
device is plugged in. But of course it can't do that if the bus
contains
other devices that are in use by the host or by other guests.
If your host OS is new enough to support vfio device assignment, I
would
suggest using that instead (if you update to the latest upstrem libvirt
it will automatically use VFIO if it is available, otherwise you can
add
"<driver name='vfio'/>" to the device definition to assure that it will
either use vfio or fail. You will also probably need to run "modprobe
vfio" before starting the guest). The reason I suggest this is that
VFIO
will automatically handle resetting the assigned devices whenever
necessary, compared to the old KVM device assignment, where libvirt
must
always reset the device because it has no better information about
whether or not it is really necessary.
Another thing you can try is plugging one of the cards into a different
slot - if you can find a slot that is on a different bus, then libvirt
will be able to reset the bus containing the card that will be assigned
(since there won't be any other active devices on that same bus).
if i (soft) remove the 2nd card via
echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove
i can start the guest with the 1st card assigned.
thx for anyone looking into this.
libvirt attempts to reset the
wbr,tja..
PS:
host:
root@father:~# lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM
Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
I217-V (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #1 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series
Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev
05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series
Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
SMBus Controller (rev 05)
02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
Bridge (rev 03)
03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev
01)
03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev
01)
root@father:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.10
DISTRIB_CODENAME=saucy
DISTRIB_DESCRIPTION="Ubuntu 13.10"
root@father:~# uname -a
Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
root@father:~# libvirtd --version
libvirtd (libvirt) 1.1.1
guest:
root@father:~# cat /etc/libvirt/qemu/tvBackend.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO
BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be
made using:
virsh edit tvBackend
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>tvBackend</name>
<uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/kvm/tvBackend.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0'
unit='0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:10:26:ec'/>
<source bridge='brlan'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
<video>
<model type='vga' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
