On Tue, May 10, 2022 at 11:25:54AM -0400, Matthew Rosato wrote: > On 5/9/22 7:35 PM, Jason Gunthorpe wrote: > > Hi s390 folks/Matthew > > > > Since everyone is looking at iommu support for the nested domains, > > could we also tackle the default domain conversion please? s390 is one > > of the last 4 drivers that need it. > > > > From what I can see it looks like when detach_dev() is called it > > expects the platform's dma_ops to work in arch/s390/pci/pci_dma.c ? > > Yes > > > > > Has anyone thought about converting the dma_ops to use the normal DMA > > API iommu support and run it through the iommu driver instead of > > through the dma_ops? > > > > Alternatively perhaps we can keep the dma_ops with some iommu > > side-change. > > It has come up before. So ultimately the goal is to be driving the dma > through the default iommu domain (via dma-iommu) rather than directly in the > dma_ops? One of our main concerns is performance loss from s390-ism > optimizations in the dma_ops like RPCIT avoidance / lazy map + > global flush The core version is somewhat different, it triggers the iotlb_flush_all from a timer, not just on address space wrap around, but the fast path on unmap can still skip the zpci_refresh_trans(). On the other hand it doesn't have the limit of iova space, and the iova allocator is somewhat more sophisticated which will optimize large page cases that s390 currently doesn't. Basically it will work better with things like mlx5 cards in the normal case. The lasy flush is done via the IOMMU_DOMAIN_DMA_FQ and the iommu gather->queued stuff to allow skipping the RCPIT during the normal iotlb_sync. > I think the reality is that Niklas and I need to have a close look and do > some testing on our end to see what it will take and if we can get > acceptable performance from a conversion, then get back to you. It would be a good long term goal, getting rid of these duplicated dma_ops is another open task. There is a patch series out there to convert arm, so this whole area will become even more niche. But another path is to somehow keep them and just allow default domains to work - ARM did this. Jason
