On Wed, Jun 05, 2013 at 10:43:17PM +0300, Michael S. Tsirkin wrote: > On Wed, Jun 05, 2013 at 01:57:16PM -0500, Anthony Liguori wrote: > > "Michael S. Tsirkin" <mst@xxxxxxxxxx> writes: > > > > > On Wed, Jun 05, 2013 at 10:46:15AM -0500, Anthony Liguori wrote: > > >> Look, it's very simple. > > > We only need to do it if we do a change that breaks guests. > > > > > > Please find a guest that is broken by the patches. You won't find any. > > > > I think the problem in this whole discussion is that we're talking past > > each other. > > > > Here is my understanding: > > > > 1) PCI-e says that you must be able to disable IO bars and still have a > > functioning device. > > > > 2) It says (1) because you must size IO bars to 4096 which means that > > practically speaking, once you enable a dozen or so PIO bars, you run > > out of PIO space (16 * 4k == 64k and not all that space can be used). > > > Let me add 3 other issues which I mentioned and you seem to miss: > > 3) architectures which don't have fast access to IO ports, exist > virtio does not work there ATM > > 4) setups with many PCI bridges exist and have the same issue > as PCI express. virtio does not work there ATM > > 5) On x86, even with nested page tables, firmware only decodes > the page address on an invalid PTE, not the data. You need to > emulate the guest to get at the data. Without > nested page tables, we have to do page table walk and emulate > to get both address and data. Since this is how MMIO > is implemented in kvm on x86, MMIO is much slower than PIO > (with nested page tables by a factor of >2, did not test without). Oh I forgot: 6) access to MMIO BARs is painful in the BIOS environment so BIOS would typically need to enable IO for the boot device. > > virtio-pci uses a IO bars exclusively today. Existing guest drivers > > assume that there is an IO bar that contains the virtio-pci registers. > > So let's consider the following scenarios: > > > > QEMU of today: > > > > 1) qemu -drive file=ubuntu-13.04.img,if=virtio > > > > This works today. Does adding an MMIO bar at BAR1 break this? > > Certainly not if the device is behind a PCI bus... > > > > But are we going to put devices behind a PCI-e bus by default? Are we > > going to ask the user to choose whether devices are put behind a legacy > > bus or the express bus? > > > > What happens if we put the device behind a PCI-e bus by default? Well, > > it can still work. That is, until we do something like this: > > > > 2) qemu -drive file=ubuntu-13.04.img,if=virtio -device virtio-rng > > -device virtio-balloon.. > > > > Such that we have more than a dozen or so devices. This works > > perfectly fine today. It works fine because we've designed virtio to > > make sure it works fine. Quoting the spec: > > > > "Configuration space is generally used for rarely-changing or > > initialization-time parameters. But it is a limited resource, so it > > might be better to use a virtqueue to update configuration information > > (the network device does this for filtering, otherwise the table in the > > config space could potentially be very large)." > > > > In fact, we can have 100s of PCI devices today without running out of IO > > space because we're so careful about this. > > > > So if we switch to using PCI-e by default *and* we keep virtio-pci > > without modifying the device IDs, then very frequently we are going to > > break existing guests because the drivers they already have no longer > > work. > > > > A few virtio-serial channels, a few block devices, a couple of network > > adapters, the balloon and RNG driver, and we hit the IO space limit > > pretty damn quickly so this is not a contrived scenario at all. I would > > expect that we frequently run into this if we don't address this problem. > > > > So we have a few options: > > 1) Punt all of this complexity to libvirt et al and watch people make > > the wrong decisions about when to use PCI-e. This will become yet > > another example of KVM being too hard to configure. > > > > 2) Enable PCI-e by default and just force people to upgrade their > > drivers. > > > > 3) Don't use PCI-e by default but still add BAR1 to virtio-pci > > > > 4) Do virtio-pcie, make it PCI-e friendly (drop the IO BAR completely), > > We can't do this - it will hurt performance. > > > give > > it a new device/vendor ID. Continue to use virtio-pci for existing > > devices potentially adding virtio-{net,blk,...}-pcie variants for > > people that care to use them. > > > > I think 1 == 2 == 3 and I view 2 as an ABI breaker. > > Why do you think 2 == 3? 2 changes default behaviour. 3 does not. > > > libvirt does like > > policy so they're going to make a simple decision and always use the > > same bus by default. I suspect if we made PCI the default, they might > > just always set the PCI-e flag just because. > > This sounds very strange. But let's assume you are right for > the sake of the argument ... > > > There are hundreds of thousands if not millions of guests with existing > > virtio-pci drivers. Forcing them to upgrade better have an extremely > > good justification. > > > > I think 4 is the best path forward. It's better for users (guests > > continue to work as they always have). There's less confusion about > > enabling PCI-e support--you must ask for the virtio-pcie variant and you > > must have a virtio-pcie driver. It's easy to explain. > > I don't think how this changes the situation. libvirt still need > to set policy and decide which device to use. > > > It also maps to what regular hardware does. I highly doubt that there > > are any real PCI cards that made the shift from PCI to PCI-e without > > bumping at least a revision ID. > > Only because the chance it's 100% compatible on the software level is 0. > It always has some hardware specific quirks. > No such excuse here. > > > It also means we don't need to play games about sometimes enabling IO > > bars and sometimes not. > > This last paragraph is wrong, it ignores the issues 3) to 5) > I added above. > > If you do take them into account: > - there are reasons to add MMIO BAR to PCI, > even without PCI express > - we won't be able to drop IO BAR from virtio > > > Regards, > > > > Anthony Liguori > > > > > > > > > > > -- > > > MST > > > -- > > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html