On Tue, Jan 11, 2011 at 11:58 AM, Prasad Joshi <prasadjoshi124@xxxxxxxxx> wrote: > On Mon, Jan 10, 2011 at 8:24 PM, Konrad Rzeszutek Wilk > <konrad.wilk@xxxxxxxxxx> wrote: >>> AMD chipset. >> >> They don't seem to have have any errata's for this GFX business. >> But they do have a passthrough mode - hopefull that is not what you have >> passed in? >> >>> > Since you mentioned that you were able to passthrough other PCI devices >>> > that means the IOMMU is working. This narrows it down. >>> >>> Yes I could pass-through a network card to VM. I also tried passing >>> through a old Sound Card to Windows 7 machine. It did not work as >>> expected, Windows 7 does not have the old drivers. >> >> Heh. >>> >>> > >>> > The big difference between other PCI devices and the graphics card >>> > is that the graphic card has its own VMM and "radeon_gart_bind" ends >>> > up programming the bus address (so virt_to_phys of the pages, and >>> > you could put in a printk there to see what it is) in the graphic >>> > VMM. This means that when the graphic card is told to fetch the >>> > writeback data it ends up using the address that was programmed >>> > in there to look. Here the IOMMU should step in and see "oh, this >>> > DMA address is for this guest, let me patch it up and actually >>> > look where this would fall in the guest space. Oh OK, let me use >>> > this value real value which correspond the guests real value." >>> >>> It will take some time for me to understand every thing you have written :) >> >> Ok. Ping me if you have some questions. Reading this might >> help (or confuse you more): http://wiki.xensource.com/xenwiki/XenPVOPSDRM >> >>> >>> > >>> > But the IOMMU has a couple of knobs. One of those is the passthrough >>> > code and the GFX mapping code. The first should be easy to spot (should >>> > tell you on bootup whether you got that enabled or not), while the other >>> > looks to be a bunch of workarounds when passing in GFX cards b/c they .. >>> > well, not sure what, but they look to not work correctly with the IOMMU. >>> >>> By GFX Mapping, >>> Do you mean the code that maps VGA frame-buffers to Guest OS? >> >> No. It is the IOMMU kernel code that checks to see if this is a GFX >> card and if so does something fancy. But the workarounds for this appear >> to be exclusive to Intel chipset so you shouldn't hit any. >> >>> >>> One more silly question on the frame-buffer mapping. >>> >>> I was reading a paper on xen vga pass-though. It says "When applying >>> PCI pass-through, certain memory areas of the physical machine are >>> mapped to the VM. When the guest OS writes to one of those memory >>> addresses, Xen will make sure it is actually written at the >>> appropriate address. This implicates that no other VM is able to make >>> use of that device. The frame-buffer is mapped to the machine's main >>> memory at address 0xA0000 to 0xC0000. This memory range must be mapped >>> to the VM's memory in order for the OS to address the video adapter." >> <nods> >>> >>> My question is.... >>> In my case, the host machine is using Nvidia graphics card, so the >>> host frame-buffer memory 0xA0000 to 0xC0000 is being actively used by >>> host. Now if the ATI card maps it's framebuffer to this same address >>> wouldn't it cause any problem. >> >> Ah, but it does not. The first card that is called by the BIOS (so your >> motherboard BIOS) sets itself up to use that memory. The other card has >> to wait. Keep in mind that the A0000 to C0000 is the old style VGA >> framebuffer (80x25) or if you program the card properly it can do >> VESA graphics, but nothing else. Usually the guest gets a Cirrus Logic >> video card passed in which sets this device up (and touching that >> area in the guest ends up in VNC window or SDL window). >> >> Anyhow back to your host.. >> If you grep in your kernel log you should see something like: >> [drm] fb mappable at 0xE0142000 >> >> This is the framebuffer address. >>> >>> If the the machine has only 1 graphics card those statement make >>> sense, but I could not understand what will happen when there are two >>> graphics cards, each connected to different monitor and assigned to >>> different VM, where would each machines frame-buffer reside in the >>> host memory. >> >> Check the kernel log. Usually it is also the first BAR in the pci device. >> >>> >>> Does Xen allow passing-though multiple graphics cards, each to different VM? >> >> Yes (with some patches posted by the AMD and Intel folks).. >> (look for threads that have some GFX or Radeon or something like that in its title). >>> >>> > >>> > If you see 'identity map for device' where the device is your GFX >>> > then that looks to be the bug. It shouldn't set the identity mapping on it. >>> > >>> >>> I guess the identity map is used only on the Intel chipset. Here are >>> msgs on host machine when addition debugging parameter >>> (amd_iommu_dump) was added. >>> >>> prasad@prasad-kvm:~$ dmesg | grep -i -e iommu -e dmar -e amd >>> [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ >>> root=/dev/sda1 ro iommu=1 amd_iommu_dump >>> [ 0.000000] ACPI: SRAT 00000000bfe9f8b0 00108 (v01 AMD FAM_F_10 >>> 00000002 AMD 00000001) >>> [ 0.000000] ACPI: IVRS 00000000bfe9fa00 000D8 (v01 AMD RD890S >>> 00202031 AMD 00000000) >>> [ 0.000000] ACPI: SSDT 00000000bfe9fae0 00DA4 (v01 A M I POWERNOW >>> 00000001 AMD 00000001) >>> [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.37-rc6+ >>> root=/dev/sda1 ro iommu=1 amd_iommu_dump >>> [ 0.000000] Please enable the IOMMU option in the BIOS setup >>> [ 0.012918] Performance Events: AMD PMU driver. >>> [ 0.137960] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00 >>> [ 0.300011] System has AMD C1E enabled >>> [ 2.680386] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300 >>> [ 2.680441] AMD-Vi: mmio-addr: 00000000f6000000 >>> [ 2.680670] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00 >>> [ 2.680726] AMD-Vi: DEV_RANGE_END devid: 00:00.2 >>> [ 2.680776] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00 >>> [ 2.680827] AMD-Vi: DEV_SELECT devid: 07:00.0 flags: 00 >>> [ 2.680883] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00 >>> [ 2.680934] AMD-Vi: DEV_SELECT devid: 06:00.0 flags: 00 >>> [ 2.680984] AMD-Vi: DEV_SELECT devid: 00:05.0 flags: 00 >>> [ 2.681035] AMD-Vi: DEV_SELECT devid: 05:00.0 flags: 00 >>> [ 2.681086] AMD-Vi: DEV_SELECT devid: 00:06.0 flags: 00 >>> [ 2.681137] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00 >>> [ 2.681187] AMD-Vi: DEV_SELECT devid: 00:07.0 flags: 00 >>> [ 2.681238] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00 >>> [ 2.681289] AMD-Vi: DEV_SELECT devid: 00:0b.0 flags: 00 >>> [ 2.681340] AMD-Vi: DEV_SELECT_RANGE_START devid: 02:00.0 flags: 00 >>> [ 2.681391] AMD-Vi: DEV_RANGE_END devid: 02:00.1 >>> [ 2.681441] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00 >>> [ 2.681492] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00 >>> [ 2.681543] AMD-Vi: DEV_RANGE_END devid: 00:12.2 >>> [ 2.681594] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00 >>> [ 2.681645] AMD-Vi: DEV_RANGE_END devid: 00:13.2 >>> [ 2.681695] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7 >>> [ 2.681746] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00 >>> [ 2.681797] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00 >>> [ 2.681847] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00 >>> [ 2.681898] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00 >>> [ 2.681949] AMD-Vi: DEV_ALIAS_RANGE devid: 01:00.0 flags: 00 >>> devid_to: 00:14.4 >>> [ 2.682010] AMD-Vi: DEV_RANGE_END devid: 01:1f.7 >>> [ 2.682855] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00 >>> [ 2.682906] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00 >>> [ 2.682957] AMD-Vi: DEV_RANGE_END devid: 00:16.2 >>> [ 2.683234] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 >>> [ 2.692043] AMD-Vi: Lazy IO/TLB flushing enabled >>> [ 4.360112] ehci_hcd 0000:00:12.2: applying AMD >>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >>> [ 4.380635] ehci_hcd 0000:00:13.2: applying AMD >>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >>> [ 4.400658] ehci_hcd 0000:00:16.2: applying AMD >>> SB700/SB800/Hudson-2/3 EHCI dummy qh workaround >>> [ 4.811320] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T >>> Processor (6 cpu cores) (version 2.20.00) >>> [ 12.349919] EDAC amd64_edac: Ver: 3.3.0 Dec 21 2010 >>> [ 12.350018] EDAC amd64: This node reports that Memory ECC is >>> currently disabled, set F3x44[22] (0000:00:18.3). >>> [ 12.350030] EDAC amd64: ECC disabled in the BIOS or no ECC >>> capability, module will not load. >>> [ 12.350044] amd64_edac: probe of 0000:00:18.2 failed with error -22 >> >> What motherboard is this? > > ASUS M4A89TD Pro/USB3, it is mentioned on the link > http://wiki.xensource.com/xenwiki/VTdHowTo > >>> >>> prasad@prasad-kvm:~$ lspci | grep -i RV >>> 02:00.0 VGA compatible controller: ATI Technologies Inc RV370 5B64 >>> [FireGL V3100 (PCIE)] (rev 80) >>> 02:00.1 Display controller: ATI Technologies Inc RV370 5B64 [FireGL >>> V3100 (PCIE)] (Secondary) (rev 80) >> >> So, how come you aren't passing in 02:00.1? > > I tried passing through 02.00.1 as well. When I pass it to a Linux > machine I do not see the device bind to a driver, I don't see the drm > logs in the dmsg. So I thought I am not supposed to pass the 02:00:1. > > Should I be passing 02.00.1 instead of 02.00.0? 02:00.0 is the actual device. 02:00.1 is just a placeholder for supporting dualhead certain other OSes. Alex > >>> >>> > Look for 'DMAR GFX' on Google. Might shed some more light. >>> > >>> >>> No DMAR for AMD. I think I should have a look at DMAR GFX. >>> >>> I am really thankful to you for your support. >> >> Hopefully I haven't confused you any further :-) >>> >>> Thanks and Regards, >>> Prasad >> > _______________________________________________ > dri-devel mailing list > dri-devel@xxxxxxxxxxxxxxxxxxxxx > http://lists.freedesktop.org/mailman/listinfo/dri-devel > _______________________________________________ dri-devel mailing list dri-devel@xxxxxxxxxxxxxxxxxxxxx http://lists.freedesktop.org/mailman/listinfo/dri-devel
