Re: kvm PCI assignment & VFIO ramblings

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 08/02/2011 04:27 AM, Benjamin Herrenschmidt wrote:
>
>  I have a feeling you'll be getting the same capabilities sooner or
>  later, or you won't be able to make use of S/R IOV VFs.

I'm not sure why you mean. We can do SR/IOV just fine (well, with some
limitations due to constraints with how our MMIO segmenting works and
indeed some of those are being lifted in our future chipsets but
overall, it works).

Don't those limitations include "all VFs must be assigned to the same guest"?

PCI on x86 has function granularity, SRIOV reduces this to VF granularity, but I thought power has partition or group granularity which is much coarser?

In -theory-, one could do the grouping dynamically with some kind of API
for us as well. However the constraints are such that it's not
practical. Filtering on RID is based on number of bits to match in the
bus number and whether to match the dev and fn. So it's not arbitrary
(but works fine for SR-IOV).

The MMIO segmentation is a bit special too. There is a single MMIO
region in 32-bit space (size is configurable but that's not very
practical so for now we stick it to 1G) which is evenly divided into N
segments (where N is the number of PE# supported by the host bridge,
typically 128 with the current bridges).

Each segment goes through a remapping table to select the actual PE# (so
large BARs use consecutive segments mapped to the same PE#).

For SR-IOV we plan to not use the M32 region. We also have 64-bit MMIO
regions which act as some kind of "accordions", they are evenly divided
into segments in different PE# and there's several of them which we can
"move around" and typically use to map VF BARs.

So, SRIOV VFs *don't* have the group limitation? Sorry, I'm deluged by technical details with no ppc background to put them to, I can't say I'm making any sense of this.

>  >
>  >  VFIO here is basically designed for one and only one thing: expose the
>  >  entire guest physical address space to the device more/less 1:1.
>
>  A single level iommu cannot be exposed to guests.  Well, it can be
>  exposed as an iommu that does not provide per-device mapping.

Well, x86 ones can't maybe but on POWER we can and must thanks to our
essentially paravirt model :-) Even if it' wasn't and we used trapping
of accesses to the table, it would work because in practice, even with
filtering, what we end up having is a per-device (or rather per-PE#
table).

>  A two level iommu can be emulated and exposed to the guest.  See
>  http://support.amd.com/us/Processor_TechDocs/48882.pdf for an example.

What you mean 2-level is two passes through two trees (ie 6 or 8 levels
right ?).

(16 or 25)

We don't have that and probably never will. But again, because
we have a paravirt interface to the iommu, it's less of an issue.

Well, then, I guess we need an additional interface to expose that to the guest.

>  >  This means:
>  >
>  >     - It only works with iommu's that provide complete DMA address spaces
>  >  to devices. Won't work with a single 'segmented' address space like we
>  >  have on POWER.
>  >
>  >     - It requires the guest to be pinned. Pass-through ->   no more swap
>
>  Newer iommus (and devices, unfortunately) (will) support I/O page faults
>  and then the requirement can be removed.

No. -Some- newer devices will. Out of these, a bunch will have so many
bugs in it it's not usable. Some never will. It's a mess really and I
wouldn't design my stuff based on those premises just yet. Making it
possible to support it for sure, having it in mind, but not making it
the fundation on which the whole API is designed.

The API is not designed around pinning. It's a side effect of how the IOMMU works. If your IOMMU only maps pages which are under active DMA, then it would only pin those pages.

But I see what you mean, the API is designed around up-front specification of all guest memory.

>  >     - It doesn't work for POWER server anyways because of our need to
>  >  provide a paravirt iommu interface to the guest since that's how pHyp
>  >  works today and how existing OSes expect to operate.
>
>  Then you need to provide that same interface, and implement it using the
>  real iommu.

Yes. Working on it. It's not very practical due to how VFIO interacts in
terms of APIs but solvable. Eventually, we'll make the iommu Hcalls
almost entirely real-mode for performance reasons.

The original kvm device assignment code was (and is) part of kvm itself. We're trying to move to vfio to allow sharing with non-kvm users, but it does reduce flexibility. We can have an internal vfio-kvm interface to update mappings in real time.

>  >  - Performance sucks of course, the vfio map ioctl wasn't mean for that
>  >  and has quite a bit of overhead. However we'll want to do the paravirt
>  >  call directly in the kernel eventually ...
>
>  Does the guest iomap each request?  Why?

Not sure what you mean... the guest calls h-calls for every iommu page
mapping/unmapping, yes. So the performance of these is critical. So yes,
we'll eventually do it in kernel. We just haven't yet.

I see. x86 traditionally doesn't do it for every request. We had some proposals to do a pviommu that does map every request, but none reached maturity.

>
>  So, you have interrupt redirection?  That is, MSI-x table values encode
>  the vcpu, not pcpu?

Not exactly. The MSI-X address is a real PCI address to an MSI port and
the value is a real interrupt number in the PIC.

However, the MSI port filters by RID (using the same matching as PE#) to
ensure that only allowed devices can write to it, and the PIC has a
matching PE# information to ensure that only allowed devices can trigger
the interrupt.

As for the guest knowing what values to put in there (what port address
and interrupt source numbers to use), this is part of the paravirt APIs.

So the paravirt APIs handles the configuration and the HW ensures that
the guest cannot do anything else than what it's allowed to.

Okay, this is something that x86 doesn't have. Strange that it can filter DMA at a fine granularity but not MSI, which is practically the same thing.

>
>  Does the BAR value contain the segment base address?  Or is that added
>  later?

It's a shared address space. With a basic configuration on p7ioc for
example we have MMIO going from 3G to 4G (PCI side addresses). BARs
contain the normal PCI address there. But that 1G is divided in 128
segments of equal size which can separately be assigned to PE#'s.

So BARs are allocated by firmware or the kernel PCI code so that devices
in different PEs don't share segments.

Okay, and config space virtualization ensures that the guest can't remap?

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux