Re: Shouldn't VFIO virtualize the ATS capability?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, 2 Dec 2016 07:45:16 +0000
Ilya Lesokhin <ilyal@xxxxxxxxxxxx> wrote:

> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx]
> > Sent: Friday, December 2, 2016 1:23 AM
> > To: Ilya Lesokhin <ilyal@xxxxxxxxxxxx>
> > Cc: linux-pci@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx; bhelgaas@xxxxxxxxxx;
> > Adi Menachem <adim@xxxxxxxxxxxx>
> > Subject: Re: Shouldn't VFIO virtualize the ATS capability?  
> ... 
> > > > > > Aren't invalidations issued by the iommu, why does the
> > > > > > hypervisor need to participate?  How would a software entity
> > > > > > induce an  
> > > > invalidation?  
> > > > > That's what one might expect from a sane design, but
> > > > > http://lxr.free-electrons.com/source/drivers/iommu/intel-iommu.c?v
> > > > > =4.8
> > > > > #L1549
> > > > > seems to imply otherwise :(  
> > 
> > This seems correct though, the device iotlb would interact with the physical
> > IOMMU, so this is happening on the host.  The call path would
> > be:
> > 
> > ioctl(container, VFIO_IOMMU_UNMAP_DMA, ...)
> >   vfio_fops_unl_ioctl
> >      vfio_iommu_type1_ioctl
> >        vfio_dma_do_unmap
> >          vfio_remove_dma
> >            vfio_unmap_unpin
> >              iommu_unmap
> >                intel_iommu_unmap
> >                  iommu_flush_iotlb_psi
> >                    iommu_flush_dev_iotlb
> > 
> > For a non-iommu VM, mappings will be mostly static, so this will be rare.  If
> > we had VT-d emulation support in the VM, the iommu domain used by the
> > VM would map to an iommu domain in the host and any invalidations within
> > that domain would trigger an unmap through to the host domain.  
> 
> My concern was for the case where the host is not aware of ATS or decides not to use it for some reason.
> In that case the guest might enable ATS and abuse the fact that the host doesn't know it needs to 
> issue invalidations to the device

Is there any valid reason that a driver would enable ATS without a
visible IOMMU?  I think we want to be careful that we're not policing
guest drivers simply because they might do something incorrectly,
especially if the incorrect behavior only affects the device.  We
really only want to hide ATS at the host level if it cannot be used
correctly or if using it incorrectly can affect devices or system
behavior outside of the realm of that user instance.  For instance, it
seems valid that a user could enable or disable ATS, but perhaps
manipulating the page size should be virtualized (we need to be careful
about not violating the spec though).  We also have the option to hide
capabilities at the QEMU level, which is a bit softer and suggests that
there are valid uses of the capability, but they may not be compatible
or necessary with the current VM instance.

Is it true that a guest driver has no business enabling ATS without an
IOMMU visible in the VM?  Preventing that case seems like the type of
scenario that vfio should not be policing, the driver is doing
something arguably wrong.  When an IOMMU is exposed to the VM, then
perhaps we have the case where the guest enabling ATS if the host has
not is a scenario where the guest cannot behave correctly.  Perhaps
we can therefore derive that vfio in the host should only expose the ATS
capability iff ATS is enabled on the host, in which case STU should
likely be virtualized.  QEMU vfio would then still have the option
whether to expose ATS to the VM, but I think the worst that could
happen would be that the guest can gratuitously disable ATS.

   
> > > > > > > 	2. Smallest Translation Unit misconfiguration. Not sure if it
> > > > > > > will cause  
> > > > > > invalid access or only poor caching behavior.  
> > 
> > I'm not sure about this either.  I think that ATS is enabled on the
> > device prior to the guest having access to it, but could the guest
> > interfere or cause poor behavior by further interaction with the ATS
> > capability.  I guess my question would be whether the guest needs
> > visibility or access to the ATS capability to still make use of it.  We
> > certainly want to take advantage of an iotlb where available.  For a
> > Linux guest we only seem to manipulate ATS enable through the iommu
> > code, so I expect a non-iommu VM to leave ATS alone.  What's the best
> > solution then, to hide the ATS capability, assuming that it works
> > transparently on the host level?  Expose it to the guest, perhaps
> > virtualizing the STU field to the VM, giving the VM enable/disable
> > control?  How can we test any of this?  Thanks,
> >   
> I don't see the benefit of exposing the capability to the guest.
> If the Host enables ATS, the guest doesn't need to take any further action to benefit from the improved caching.
> If the host doesn't enable it, he won't issue invalidation either, so allowing the guest to enable it is unsafe.

So perhaps the right answer is that host vfio should hide the ATS
capability unless ATS is enabled by the host and virtualize the STU to
prevent the user programming conflicting values.  The assumption being
that it's never valid for the user to enable ATS if the host has not,
but it is valid for the user to be able to disable/re-enable (perhaps
for performance testing or debug).  With such a change, is there any
additional value in QEMU further hiding ATS from the guest?  We could
prevent gratuitous disables, but we don't know that there's any need
to do so.  Thanks,

Alex 
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux