Lennert Buytenhek <buytenh <at> wantstofly.org> writes: > > Hi! > > Some time ago already, we got graphics card pci passthrough working > on a friend's computer, an ASRock Z68 Extreme4 Gen3 based PC with > an Intel i7-2600, running on a Linux 3.2 kernel, with two Gigabyte > GV-R677SL-1GD (AMD Radeon HD 6770) graphics cards in CrossFireX mode. > > To get things to work we had to: > > - Boot with intel_iommu=yes. This broke the onboard Marvell > 88SE9120 controller (an AHCI controller) with IOMMU related > kernel errors, and we had to move some drives around. > > - Blacklist the radeon audio driver so that the graphics cards > aren't in use on the host. > > - Apply the patches at the end of this mail to kvm and SeaBIOS to > allow for more BAR space under 4G. (The relevant BARs on the > graphics cards _are_ 64 bit BARs, but kvm seemed to turn those > into 32 bit BARs in the guest.) > > - Apply the hacky patch at the end of this mail to SeaBIOS to > always skip initialising the Radeon's option ROMs, or the VM > would hang inside the Radeon option ROM if you boot the VM > without the default cirrus video. > > With these patches, the Windows 7 guest seemed to work OK on basic > testing, allowing playing of all 3d games we tried, including in > CrossFireX mode. We ran into the following issues: > > - The AMD Catalyst graphics card drivers bluescreen the VM on win7 > boot if you boot with a CPU family/model/stepping that it does not > like and one or more Radeon cards passed through to the VM. > (Setting the guest family/model/stepping to be equal to the host's > CPU's fixes the bluescreen and allows booting to complete.) > > - AMD Catalyst 12.4 worked okay, but the 12.6 beta would bluescreen > the VM. > > - Playing video in Flash (e.g. youtube) and XBMC gives a green > rectangle where the video is supposed to be. This was fixable in > both cases by turning off hardware acceleration (in Flash) or > "dxva2" (in xbmc). A problem with these symptoms is noted in the > AMD Catalyst driver release notes, but the same problem does not > seem to appear when running Windows 7 directly on the host. > (Could this be related to skipping the option ROM?) > > We also tried passing through an NVidia graphics card (GTX 260) > instead of the Radeons, but that would crash the VM the couple of > times we tried, and we didn't look into this further. > > The setup (display on the right assigned to the Linux host, display > on the left assigned to the Windows guest): http://i.imgur.com/idtVH.jpg > > 3DMark running on Windows host: http://i.imgur.com/nvbQr.jpg > > 3DMark running on Windows as a VM: http://i.imgur.com/vuL7H.jpg > > (Running with only one graphics card gives about half the 3dmark > score, so CrossFireX does appear to be working in both cases.) > > cheers, > Lennert > > KVM patch for more <4G PCI BAR space: > > diff -ur qemu-kvm-1.0+noroms/hw/pc_piix.c ../qemu-kvm-1.0+noroms/hw/pc_piix.c > --- qemu-kvm-1.0+noroms/hw/pc_piix.c 2011-12-04 11:38:06.000000000 +0100 > +++ ../qemu-kvm-1.0+noroms/hw/pc_piix.c 2012-06-09 16:24:14.186327304 +0200 > @@ -112,9 +112,9 @@ > kvmclock_create(); > } > > - if (ram_size >= 0xe0000000 ) { > - above_4g_mem_size = ram_size - 0xe0000000; > - below_4g_mem_size = 0xe0000000; > + if (ram_size >= 0xc0000000 ) { > + above_4g_mem_size = ram_size - 0xc0000000; > + below_4g_mem_size = 0xc0000000; > } else { > above_4g_mem_size = 0; > below_4g_mem_size = ram_size; > > Corresponding SeaBIOS patch: > > diff --git a/src/acpi-dsdt.dsl b/src/acpi-dsdt.dsl > index 15299ee..31e7102 100644 > --- a/src/acpi-dsdt.dsl > +++ b/src/acpi-dsdt.dsl > @@ -170,10 +170,10 @@ DefinitionBlock ( > ,, , AddressRangeMemory, TypeStatic) > DWordMemory (ResourceProducer, PosDecode, MinFixed, MaxFixed, NonCacheable, ReadWrite, > 0x00000000, // Address Space Granularity > - 0xE0000000, // Address Range Minimum > + 0xC0000000, // Address Range Minimum > 0xFEBFFFFF, // Address Range Maximum > 0x00000000, // Address Translation Offset > - 0x1EC00000, // Address Length > + 0x3EC00000, // Address Length > ,, PW32, AddressRangeMemory, TypeStatic) > }) > Name (CR64, ResourceTemplate () > diff --git a/src/config.h b/src/config.h > index 3a70867..b305d9a 100644 > --- a/src/config.h > +++ b/src/config.h > @@ -44,9 +44,9 @@ > #define BUILD_SMM_INIT_ADDR 0x38000 > #define BUILD_SMM_ADDR 0xa8000 > #define BUILD_SMM_SIZE 0x8000 > -#define BUILD_MAX_HIGHMEM 0xe0000000 > +#define BUILD_MAX_HIGHMEM 0xc0000000 > > -#define BUILD_PCIMEM_START 0xe0000000 > +#define BUILD_PCIMEM_START 0xc0000000 > #define BUILD_PCIMEM_END 0xfec00000 /* IOAPIC is mapped at */ > #define BUILD_PCIMEM64_START 0x8000000000ULL > #define BUILD_PCIMEM64_END 0x10000000000ULL > > SeaBIOS patch to always skip Radeon option ROM initialization: > > diff --git a/src/optionroms.c b/src/optionroms.c > index 00697b2..857956a 100644 > --- a/src/optionroms.c > +++ b/src/optionroms.c > @@ -440,6 +440,8 @@ vga_setup(void) > foreachpci(pci) { > if (!is_pci_vga(pci)) > continue; > + if (pci->vendor == 0x1002 && pci->device == 0x68ba) > + continue; > vgahook_setup(pci); > init_pcirom(pci, 1, NULL); > break; > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo <at> vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hi There, I just wanted to post a follow up to this to say I've also managed to get a reasonably stable system with a similar setup. Hardware: - asrock q77m vPro (with vt-d enabled) - core i7 3770 - sapphire radeon 7850 2gb Software inside guest - windows 8 64bit - standard/home edition - radeon catalyst drivers 12-10_vista_win7_win8_64_dd_cc_whql_net4 host OS: - ubuntu 12.10, booted with intel_iommu=yes - blacklisted radeon - stock kvm, libvirt, qemu - seabios recompiled to latest and including the below optionroms.c patch (modified for me device id), not sure if the patch has made any difference though. - sapphire rom downloaded from http://www.techpowerup.com/vgabios/ - libvirt / qemu config updated to include the rom in the PCI passthrough section: <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/etc/libvirt/roms/Sapphire.HD7850.2048.120306_1.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> What i've observed: * my original configuration used the stock seabios and without the rom file directive, this would mostly startup windows 8 fine from a cold boot of the host. restarting win8 would lead to the host locking up completely most of the time, nothing reported on the console for the kernel. * as above, but shutting down windows 8 and then restarting it would work for about the first three attempts, the fourth would cause the vm to goto 100% cpu, powering off the vm and restarting would lock the host After applying the seabios change and including the rom file the system appears to be much more stable. I have managed to shutdown and restart win8 a number of times without rebooting the host. Probably so far tried about 6 or 7 shutdown/starts in a row. again with the seabios and rom file changes, restarting windows8 seems to cause lockups. HTH, -vince -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
