On Wed, 2023-09-27 at 13:24 +0200, Niklas Schnelle wrote: > On Wed, 2023-09-27 at 11:55 +0200, Joerg Roedel wrote: > > Hi Niklas, > > > > On Wed, Sep 27, 2023 at 10:55:23AM +0200, Niklas Schnelle wrote: > > > The problem is that something seems to be broken in the iommu/core > > > branch. Regardless of whether I have my DMA API conversion on top or > > > with the base iommu/core branch I can not use ConnectX-4 VFs. > > > > Have you already tried to bisect the issue in the iommu/core branch? > > The result might sched some light on the issue. > > > > Regards, > > > > Joerg > > Hi Joerg, > > Working on it, somehow I must have messed up earlier. It now looks like > it might in fact be caused by my DMA API conversion rebase and the > "s390/pci: Use dma-iommu layer" commit. Maybe there is some interaction > with Jason's patches that I haven't thought about. So sorry for any > wrong blame. > > Thanks, > Niklas Hi, I tracked the problem down from mlx5_core's alloc_cmd_page() via dma_alloc_coherent(), ops->alloc, iommu_dma_alloc_remap(), and __iommu_dma_alloc_noncontiguous() to a failed iommu_dma_alloc_iova(). The allocation here is for 4K so nothing crazy. On second look I also noticed: nvme 2007:00:00.0: Using 42-bit DMA addresses for the NVMe that is working. The problem here seems to be that we set iommu_dma_forcedac = true in s390_iommu_probe_finalize() because we have currently have a reserved region over the first 4 GiB anyway so will always use IOVAs larger than that. That however is too late since iommu_dma_set_pci_32bit_workaround() is already checked in __iommu_probe_device() which is called just before ops- >probe_finalize(). So I moved setting iommu_dma_forcedac = true to zpci_init_iommu() and that gets rid of the notice for the NVMe but I still get a failure of iommu_dma_alloc_iova() in __iommu_dma_alloc_noncontiguous(). So I'll keep digging. Thanks, Niklas
