On 2011-05-09 17:48, Alex Williamson wrote:
> On Mon, 2011-05-09 at 17:27 +0200, Jan Kiszka wrote:
>> On 2011-05-09 16:55, Prasad Joshi wrote:
>>> On Mon, May 9, 2011 at 12:14 PM, Jan Kiszka <jan.kiszka@xxxxxxxxxxx> wrote:
>>>> On 2011-05-05 17:17, Alex Williamson wrote:
>>>>>> And what about the host? When does Linux release the legacy range?
>>>>>> Always or only when a specific (!=vga/vesa) framebuffer driver is loaded?
>>>>>
>>>>> Well, that's where it'd be nice if the vga arbiter was actually in more
>>>>> widespread use. It currently seems to be nothing more than a shared
>>>>> mutex, but it would actually be useful if it included backends to do the
>>>>> chipset vga routing changes. I think when I was testing this, I was
>>>>> externally poking PCI bridge chipset to toggle the VGA_EN bit.
>>>>
>>>> Right, we had to drop the approach to pass through the secondary card
>>>> for now, the arbiter was not switching properly. Haven't checked yet if
>>>> VGA_EN was properly set, though the kernel code looks like it should
>>>> take care of this.
>>>>
>>>> Even with handing out the primary adapter, we had only mixed success so
>>>> far. The onboard adapter worked well (in VESA mode), but the NVIDIA is
>>>> not displaying early boot messages at all. Maybe a vgabios issue.
>>>> Windows was booting nevertheless - until we installed the NVIDIA
>>>> drivers. Than it ran into a blue screen.
>>>>
>>>> BTW, what ATI adapter did you use precisely, and what did work, what not?
>>>
>>> Not hijacking the mail thread. Just wanted to provide some inputs.
>>
>> Much appreciated in fact!
>>
>>>
>>> Few days back I had tried passing through the secondary graphics card.
>>> I could pass-through two graphics cards to virtual machine.
>>>
>>> 02:00.0 VGA compatible controller: ATI Technologies Inc Redwood
>>> [Radeon HD 5670] (prog-if 00 [VGA controller])
>>> Subsystem: PC Partner Limited Device e151
>>> Flags: bus master, fast devsel, latency 0, IRQ 87
>>> Memory at d0000000 (64-bit, prefetchable) [size=256M]
>>> Memory at fe6e0000 (64-bit, non-prefetchable) [size=128K]
>>> I/O ports at b000 [size=256]
>>> Expansion ROM at fe6c0000 [disabled] [size=128K]
>>> Capabilities: <access denied>
>>> Kernel driver in use: radeon
>>> Kernel modules: radeon
>>>
>>> 07:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS
>>> 290] (rev a1) (prog-if 00 [VGA controller])
>>> Subsystem: nVidia Corporation Device 0492
>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>>> ParErr-Stepping- SERR+ FastB2B- DisINTx-
>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>>>> TAbort-<TAbort- <MAbort- >SERR- <PERR- INTx-
>>> Latency: 0, Cache Line Size: 64 bytes
>>> Interrupt: pin A routed to IRQ 24
>>> Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
>>> Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
>>> Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
>>> Region 5: I/O ports at ec00 [size=128]
>>> Expansion ROM at fe9e0000 [disabled] [size=128K]
>>> Capabilities: <access denied>
>>> Kernel driver in use: nouveau
>>> Kernel modules: nouveau, nvidiafb
>>>
>>> Both of them are PCIe cards. I have one more ATI card and another
>>> NVIDIA card which does not work.
>>
>> Interesting. That may rule out missing PCIe capabilities as source for
>> the NVIDIA driver indisposition.
>>
>> Did you passed those cards each as primary to the guest, or was the
>> guest seeing multiple adapters? I presume you only got output after
>> early boot was completed, right?
>>
>> To avoid having to deal with legacy I/O forwarding, we started with a
>> dual adapter setup in the hope to leave the primary guest adapter at
>> know-to-work cirrus-vga. But already in a native setup with on-board
>> primary + NVIDIA secondary, the NVIDIA Windows drivers refused to talk
>> to its hardware in this constellation.
>>
>>>
>>> One of the reason the pass-through did not work is because of the
>>> limit on amount of pci configuration memory by SeaBIOS. SeaBIOS places
>>> a hard limit of 256MB or so on the amount of PCI memory space. Thus,
>>> for some of the VGA device that need more memory never worked for me.
>>>
>>> SeaBIOS allows this memory region to be extended to some value near
>>> 512MB, but even then the range is not enough.
>>>
>>> Another problem with SeaBIOS which limits the amount of memory space
>>> is: SeaBIOS allocates the BAR regions as they are encountered. As far
>>> as I know, the BAR regions should be naturally aligned. Thus the
>>> simple strategy of the SeaBIOS results in large fragmentation.
>>> Therefore, even after increasing the PCI memory space to 512MB the BAR
>>> regions were unallocated.
>>
>> That's an interesting trace! We'll check this here, but I bet it
>> contributes to the problems. Our FX 3800 has 1G memory...
>
> Yes, qemu leaves far too little MMIO space to think about assigning
> graphics cards. Both of my cards have 512MB and I hacked qemu to leave
> a bigger gap via something like:
>
> diff --git a/hw/pc.c b/hw/pc.c
> index 0ea6d10..a6376f8 100644
> --- a/hw/pc.c
> +++ b/hw/pc.c
> @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model)
> }
> }
>
> +#define PC_MAX_LOW_RAM 0xc0000000
> +
> void pc_memory_init(ram_addr_t ram_size,
> const char *kernel_filename,
> const char *kernel_cmdline,
> @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size,
> int bios_size, isa_bios_size;
> void *fw_cfg;
>
> - if (ram_size >= 0xe0000000 ) {
> - above_4g_mem_size = ram_size - 0xe0000000;
> - below_4g_mem_size = 0xe0000000;
> + if (ram_size >= PC_MAX_LOW_RAM ) {
> + above_4g_mem_size = ram_size - PC_MAX_LOW_RAM;
> + below_4g_mem_size = PC_MAX_LOW_RAM;
> } else {
> below_4g_mem_size = ram_size;
> }
>
> There's also a #define that needs to be changed in seabios config.h and
> and acpi dsdt update, but I can't seem to find patches for those.
Hmm, as this does not scale with the constantly growing memory sizes of
GPUs, I guess this should would for us as well, even with 1G. The
adapters likely only map a window to their on-board RAM.
> Also
> pay attention to the cpu_register_physical_memory calls in
> i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO
> range from you. I just commented them out:
>
> diff --git a/hw/piix_pci.c b/hw/piix_pci.c
> index b5589b9..1327563 100644
> --- a/hw/piix_pci.c
> +++ b/hw/piix_pci.c
> @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d)
> }
> smram = d->dev.config[I440FX_SMRAM];
> if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) {
> - cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000);
> + //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000);
> } else {
> for(addr = 0xa0000; addr < 0xc0000; addr += 4096) {
> - cpu_register_physical_memory(addr, 4096,
> - d->isa_page_descs[(addr - 0xa0000) >> 12]);
> + //cpu_register_physical_memory(addr, 4096,
> + // d->isa_page_descs[(addr - 0xa0000) >> 12]);
> }
> }
> }
>
> That's all the tricks I remember. Thanks,
Yeah, we are already carrying half of the above in our tree (only the
second disabling is actually needed, KVM does not support SMM). I
started looking into fixing PAM/SMRAM mess, but it's not yet beautiful -
partly because we urgently need slot management at core level.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
