On Wed, 2023-05-24 at 16:53 +0200, Niklas Schnelle wrote: > In some virtualized environments, including s390 paged memory guests, > IOTLB flushes are used to update IOMMU shadow tables. Due to this, they > are much more expensive than in typical bare metal environments or > non-paged s390 guests. In addition they may parallelize poorly in > virtualized environments. This changes the trade off for flushing IOVAs > such that minimizing the number of IOTLB flushes trumps any benefit of > cheaper queuing operations or increased paralellism. > > In this scenario per-CPU flush queues pose several problems. Firstly > per-CPU memory is often quite limited prohibiting larger queues. > Secondly collecting IOVAs per-CPU but flushing via a global timeout > reduces the number of IOVAs flushed for each timeout especially on s390 > where PCI interrupts may not be bound to a specific CPU. > > Let's introduce a single flush queue mode that reuses the same queue > logic but only allocates a single global queue. This mode is selected by > dma-iommu if a newly introduced .shadow_on_flush flag is set in struct > dev_iommu. As a first user the s390 IOMMU driver sets this flag during > probe_device. With the unchanged small FQ size and timeouts this setting > is worse than per-CPU queues but a follow up patch will make the FQ size > and timeout variable. Together this allows the common IOVA flushing code > to more closely resemble the global flush behavior used on s390's > previous internal DMA API implementation. > > Link: https://lore.kernel.org/linux-iommu/3e402947-61f9-b7e8-1414-fde006257b6f@xxxxxxx/ > Reviewed-by: Matthew Rosato <mjrosato@xxxxxxxxxxxxx> #s390 > Signed-off-by: Niklas Schnelle <schnelle@xxxxxxxxxxxxx> > --- Gentle ping. As stated in the cover letter this version aimed to resolve the only outstanding feedback of removing the .tune_dma_iommu() op in favor of a .shadow_on_flush flag in struct dev_iommu. This then let's the dma-iommu choose a single queue and larger timeouts and IOVA counts. This leaves the dma-iommu with full responsibility for the settings. Thanks, Niklas
