On Wed, Aug 14, 2013 at 11:14:48AM -0600, Alex Williamson wrote: > On Wed, 2013-08-14 at 17:47 +0300, Ville Syrjälä wrote: > > On Wed, Aug 14, 2013 at 07:23:57AM -0600, Alex Williamson wrote: > > > Hi, > > > > > > I'm trying to add support for device assignment of PCI VGA devices with > > > VFIO and QEMU. For normal, discrete discrete graphics the Linux VGA > > > arbiter works fairly well, disabling VGA on one bridge and adding it to > > > another (though I wish all the kernel VGA drivers made use of it). The > > > i915 driver only seems to support disabling VGA on really old GMCH > > > devices (see intel_modeset_vga_set_state). This means that if I boot > > > with IGD as the primary graphics and attempt to assign a discrete > > > graphics device, all the VGA range accesses are still routed to IGD, I > > > end up getting some error messages from the IGD interrupt handler, and > > > the discrete card never initializes. > > > > > > I spent some time looking through the Sand Bridge, Ivy Bridge, and > > > Haswell datasheets, and I'm a bit concerned whether the hardware even > > > provides a reasonable way to disable VGA anymore. Quoting 2.17 from the > > > Haswell docs: > > > > > > Accesses to the VGA memory range are directed to IGD depend on > > > the configuration. The configuration is specified by: > > > * Internal graphics controller in Device 2 is enabled > > > (DEVEN.D2EN bit 4) > > > * Internal graphics VGA in Device 0 Function 0 is enabled > > > through register GGC bit 1. > > > * IGD's memory accesses (PCICMD2 04 – 05h, MAE bit 1) in > > > Device 2 configuration space are enabled. > > > * VGA compatibility memory accesses (VGA Miscellaneous > > > Output register – MSR Register, bit 1) are enabled. > > > * Software sets the proper value for VGA Memory Map Mode > > > register (VGA GR06 Register, bits 3-2). See the > > > following table for translations. > > > > > > (There's a similar list for VGA I/O range) I've found that if I disable > > > memory and I/O in the PCI command register for IGD then I do get VGA > > > routing to the PEG device and the discrete VBIOS works. This obviously > > > isn't a good option for the VGA arbiter since it entirely disables IGD. > > > > > > The GGC registers aren't meant for runtime switching and are actually > > > locked. Disabling IGD via the device 2 enable bit doesn't seem like and > > > option. I don't quite understand the VGA miscellaneous output register > > > and VGA memory map mode, but the table provided for the latter makes me > > > think they just augment the VGA ranges and don't disable them. > > > > Bit 1 of MSR (0x3c2/0x3cc) should allow you to turn off VGA mem > > access while leaving other memory space access working. > > > > As for VGA I/O decode, IIRC there's no standard bit for that in VGA > > or PCI config registers, and I can't see any other bit for it in the > > docs. But I guess you could just turn off I/O space completely > > via the PCI_COMMAND register. We shouldn't need it for anything beyond > > i915_disable_vga() and that has the necessary vgaarb calls already. > > Thanks Ville. The MSR seems to work for VGA memory. Disabling I/O via > PCI_COMMAND does works, but something is re-enabling it after > intel_modeset_vga_set_state(). If I manually disable I/O with setpci > then I do have VGA routing to PEG and can still interact with the KMS > console on IGD. It's unfortunate that the MSR bit for I/O only disables > pieces of the range. If we have no other options, I'll try to hunt down > where I/O is being re-enabled and see how feasible it is to avoid. > Thanks, Hmm. Now that I look at vgaarb more it seems I misunderstood the way it works. Based on the code it looks like it will permanently remove the device from the arbiration if set_vga_decode indicates that it doesn't decode legacy resources. And it calls set_vga_decode w/ decode=false if there are more than two VGA cards in the system. That means i915_disable_vga() is actually broken whenever there is another VGA card in the system. To make it work I think what we'd need to do is always return VGA_RSRC_LEGACY_IO from i915_vga_set_decode(), which will leave the PCI_COMMAND I/O bit in the hands of vgaarb, and then poke at the MSR register to disable the VGA mem decode permanently. But to touch MSR we actually need VGA I/O, so I guess we'd have to do that right after registering w/ vgaarb. Doing it from i915_vga_set_decode() doesn't look possible since there's no guarantee that VGA I/O would end up at the right device at the time that is called. So maybe the following patch might work (although maybe vgaarb itself should be extended a bit to properly support this use case).
