On 04/02/2013 03:32:17 PM, Alex Williamson wrote:
On Tue, 2013-04-02 at 17:32 +0000, Yoder Stuart-B08248 wrote:
> 2. MSI window mappings
>
> The more problematic question is how to deal with MSIs. We need
to
> create mappings for up to 3 MSI banks that a device may need to
target
> to generate interrupts. The Linux MSI driver can allocate MSIs
from
> the 3 banks any way it wants, and currently user space has no
way of
> knowing which bank may be used for a given device.
>
> There are 3 options we have discussed and would like your
direction:
>
> A. Implicit mappings -- with this approach user space would not
> explicitly map MSIs. User space would be required to set the
> geometry so that there are 3 unused windows (the last 3
windows)
> for MSIs, and it would be up to the kernel to create the
mappings.
> This approach requires some specific semantics (leaving 3
windows)
> and it potentially gets a little weird-- when should the
kernel
> actually create the MSI mappings? When should they be
unmapped?
> Some convention would need to be established.
VFIO would have control of SET/GET_ATTR, right? So we could reduce
the
number exposed to userspace on GET and transparently add MSI entries
on
SET.
What do you mean by "reduce the number exposed"? Userspace decides how
many entries there are, but it must be a power of two beteen 1 and 256.
On x86 the interrupt remapper handles this transparently when MSI
is enabled and userspace never gets direct access to the device MSI
address/data registers.
x86 has a totally different mechanism here, as far as I understand --
even before you get into restrictions on mappings.
What kind of restrictions do you have around
adding and removing windows while the aperture is enabled?
Subwindows can be modified while the aperture is enabled, but the
aperture size and number of subwindows cannot be changed.
> B. Explicit mapping using DMA map flags. The idea is that a new
> flag to DMA map (VFIO_DMA_MAP_FLAG_MSI) would mean that
> a mapping is to be created for the supplied iova. No vaddr
> is given though. So in the above example there would be a
> a dma map at 0x10000000 for 24KB (and no vaddr). It's
> up to the kernel to determine which bank gets mapped where.
> So, this option puts user space in control of which windows
> are used for MSIs and when MSIs are mapped/unmapped. There
> would need to be some semantics as to how this is used-- it
> only makes sense
This could also be done as another "type2" ioctl extension.
Again, what is "type2", specifically? If someone else is adding their
own IOMMU that is kind of, sort of like PAMU, how would they know if
it's close enough? What assumptions can a user make when they see that
they're dealing with "type2"?
What's the value to userspace in determining which windows are used
by which banks?
That depends on who programs the MSI config space address. What is
important is userspace controlling which iovas will be dedicated to
this, in case it wants to put something else there.
It sounds like the case that there are X banks and if userspace wants
to
use MSI it needs to leave X windows available for that. Is this just
buying userspace a few more windows to allow them the choice between
MSI
or RAM?
Well, there could be that. But also, userspace will generally have a
much better idea of the type of mappings it's creating, so it's easier
to keep everything explicit at the kernel/user interface than require
more complicated code in the kernel to figure things out automatically
(not just for MSIs but in general).
If the kernel automatically creates the MSI mappings, when does it
assume that userspace is done creating its own? What if userspace
doesn't need any DMA other than the MSIs? What if userspace wants to
continue dynamically modifying its other mappings?
> C. Explicit mapping using normal DMA map. The last idea is that
> we would introduce a new ioctl to give user-space an fd to
> the MSI bank, which could be mmapped. The flow would be
> something like this:
> -for each group user space calls new ioctl
VFIO_GROUP_GET_MSI_FD
> -user space mmaps the fd, getting a vaddr
> -user space does a normal DMA map for desired iova
> This approach makes everything explicit, but adds a new ioctl
> applicable most likely only to the PAMU (type2 iommu).
And the DMA_MAP of that mmap then allows userspace to select the
window
used? This one seems like a lot of overhead, adding a new ioctl, new
fd, mmap, special mapping path, etc.
There's going to be special stuff no matter what. This would keep it
separated from the IOMMU map code.
I'm not sure what you mean by "overhead" here... the runtime overhead
of setting things up is not particularly relevant as long as it's
reasonable. If you mean development and maintenance effort, keeping
things well separated should help.
-Scott
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
