On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> wrote: >> AMD chipset. > > They don't seem to have have any errata's for this GFX business. > But they do have a passthrough mode - hopefull that is not what you have > passed in? > >> > Since you mentioned that you were able to passthrough other PCI devices >> > that means the IOMMU is working. This narrows it down. >> >> Yes I could pass-through a network card to VM. I also tried passing >> through a old Sound Card to Windows 7 machine. It did not work as >> expected, Windows 7 does not have the old drivers. > > Heh. >> >> > >> > The big difference between other PCI devices and the graphics card >> > is that the graphic card has its own VMM and "radeon_gart_bind" ends >> > up programming the bus address (so virt_to_phys of the pages, and >> > you could put in a printk there to see what it is) in the graphic >> > VMM. This means that when the graphic card is told to fetch the >> > writeback data it ends up using the address that was programmed >> > in there to look. Here the IOMMU should step in and see "oh, this >> > DMA address is for this guest, let me patch it up and actually >> > look where this would fall in the guest space. Oh OK, let me use >> > this value real value which correspond the guests real value." >> >> It will take some time for me to understand every thing you have written :) > > Ok. Ping me if you have some questions. Reading this might > help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM > >> >> > >> > But the IOMMU has a couple of knobs. One of those is the passthrough >> > code and the GFX mapping code. The first should be easy to spot (should >> > tell you on bootup whether you got that enabled or not), while the other >> > looks to be a bunch of workarounds when passing in GFX cards b/c they .. >> > well, not sure what, but they look to not work correctly with the IOMMU. >> >> By GFX Mapping, >> Do you mean the code that maps VGA frame-buffers to Guest OS? > > No. It is the IOMMU kernel code that checks to see if this is a GFX > card and if so does something fancy. But the workarounds for this appear > to be exclusive to Intel chipset so you shouldn't hit any. > >> >> One more silly question on the frame-buffer mapping. >> >> I was reading a paper on xen vga pass-though. It says "When applying >> PCI pass-through, certain memory areas of the physical machine are >> mapped to the VM. When the guest OS writes to one of those memory >> addresses, Xen will make sure it is actually written at the >> appropriate address. This implicates that no other VM is able to make >> use of that device. The frame-buffer is mapped to the machine's main >> memory at address 0xA0000 to 0xC0000. This memory range must be mapped >> to the VM's memory in order for the OS to address the video adapter." > <nods> >> >> My question is.... >> In my case, the host machine is using Nvidia graphics card, so the >> host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by >> host. Now if the ATI card maps it's framebuffer to this same address >> wouldn't it cause any problem. > > Ah, but it does not. The first card that is called by the BIOS (so your > motherboard BIOS) sets itself up to use that memory. The other card has > to wait. Keep in mind that the A0000 to C0000 is the old style VGA > framebuffer (80x25) or if you program the card properly it can do > VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic > video card passed in which sets this device up (and touching that > area in the guest ends up in VNC window or SDL window). > > Anyhow back to your host.. > If you grep in your kernel log you should see something like: > [drm] fb mappable at 0xE0142000 > > This is the framebuffer address. >> >> If the the machine has only 1 graphics card those statement make >> sense, but I could not understand what will happen when there are two >> graphics cards, each connected to different monitor and assigned to >> different VM, where would each machines frame-buffer reside in the >> host memory. > > Check the kernel log. Usually it is also the first BAR in the pci device. > >> >> Does Xen allow passing-though multiple graphics cards, each to different VM? > > Yes (with some patches posted by the AMD and Intel folks).. > (look for threads that have some GFX or Radeon or something like that in its title). >> >> > >> > If you see 'identity map for device' where the device is your GFX >> > then that looks to be the bug. It shouldn't set the identity mapping on it. >> > >> >> I guess the identity map is used only on the Intel chipset. Here are >> msgs on host machine when addition debugging parameter >> (amd_iommu_dump) was added. >> >> prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd >> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ >> root=/dev/sda1 ro iommu=1 amd_iommu_dump >> [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 >> 00000002 AMD 00000001) >> [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S >> 00202031 AMD 00000000) >> [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW >> 00000001 AMD 00000001) >> [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ >> root=/dev/sda1 ro iommu=1 amd_iommu_dump >> [ 0.000000] Please enable the IOMMU option in the BIOS setup >> [ 0.012918] Performance Events: AMD PMU driver. >> [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 >> [ 0.300011] System has AMD C1E enabled >> [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 >> [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 >> [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 >> [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 >> [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 >> [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 >> [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 >> [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 >> [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 >> [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 >> [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 >> [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 >> [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 >> [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 >> [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 >> [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 >> [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 >> [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 >> [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 >> [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 >> [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 >> [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 >> [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 >> [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 >> [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 >> [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 >> [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 >> [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 >> devid_to: 00:14.4 >> [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 >> [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 >> [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 >> [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 >> [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 >> [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled >> [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD >> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >> [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD >> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >> [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD >> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >> [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T >> Processor (6 cpu cores) (version 2.20.00) >> [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 >> [ 12.350018] EDAC amd64: This node reports that Memory ECC is >> currently disabled, set F3x44[22] (0000:00:18.3). >> [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC >> capability, module will not load. >> [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22 > > What motherboard is this? ASUS M4A89TD Pro/USB3, it is mentioned on the link http://wiki.xensource.com/xenwiki/VTdHowTo >> >> prasad@prasad-kvm:~$ lspci | grep -i RV >> 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 >> [FireGL V3100 (PCIE)] (rev 80) >> 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL >> V3100 (PCIE)] (Secondary) (rev 80) > > So, how come you aren't passing in 02:00.1? I tried passing through 02.00.1 as well. When I pass it to a Linux machine I do not see the device bind to a driver, I don't see the drm logs in the dmsg. So I thought I am not supposed to pass the 02:00:1. Should I be passing 02.00.1 instead of 02.00.0? >> >> > Look for 'DMAR GFX' on Google. Might shed some more light. >> > >> >> No DMAR for AMD. I think I should have a look at DMAR GFX. >> >> I am really thankful to you for your support. > > Hopefully I haven't confused you any further :-) >> >> Thanks and Regards, >> Prasad > _______________________________________________ dri-devel mailing list dri-devel@xxxxxxxxxxxxxxxxxxxxx http://lists.freedesktop.org/mailman/listinfo/dri-devel
