Hi Major, Thanks for the report! On Wed, Dec 21, 2022 at 08:38:46PM +0530, Major Saheb wrote: > I have an ubuntu guest running on kvm , and I am passing it 10 qemu > emulated nvme drives > <iommu model='intel'> > <driver intremap='on' eim='on'/> > </iommu> > <qemu:arg value='pcie-root-port,id=pcie-root-port%d,slot=%d'/> > <qemu:arg value='nvme,drive=NVME%d,serial=%s_%d,id=NVME%d,bus=pcie-root-port%d'/> > > kernel > Linux node-1 5.15.0-56-generic #62-Ubuntu SMP ----- x86_64 x86_64 > x86_64 GNU/Linux > > kernel command line > intel_iommu=on > > I have attached these drives to vfio-pcie. > > when I try to send IO commands to these drives VIA a userspace nvme > driver using VFIO I get > [ 1474.752590] DMAR: DRHD: handling fault status reg 2 > [ 1474.754463] DMAR: [DMA Read NO_PASID] Request device [0b:00.0] > fault addr 0xffffe000 [fault reason 0x06] PTE Read access is not set > > Can someone explain to me what's happening here ? I'm not an IOMMU expert, but I think the device (0b:00.0, I assume an nvme device) did a DMA read to 0xffffe000 (which looks suspiciously like a null pointer (-8192 off a null pointer)), and the IOMMU had no mapping for that address. Can you point us to the userspace nvme driver? I'm not a VFIO expert either, but I assume it uses something like a VFIO_IOMMU_MAP_DMA ioctl to map buffers and get IOVAs to give to the device? Can you collect a dmesg log and output of "sudo lspci -vv" for your guest? Is this something that worked in the past and broke on a newer kernel? It looks like you're using a 5.15 kernel; have you tried any newer kernels? Bjorn
