Thanks for the pointers , It seems there was some issue in QEMU emulator version 6.2.0, I updated it to QEMU emulator version 7.1.92 and the issue was gone. I don't dig deeper though to find out exactly what in QEMU. On Sat, Dec 31, 2022 at 4:06 PM David Woodhouse <dwmw2@xxxxxxxxxxxxx> wrote: > > > > On 31 December 2022 10:13:37 GMT, Yi Liu <yi.l.liu@xxxxxxxxx> wrote: > >On 2022/12/31 04:07, David Woodhouse wrote: > >> > >> > >> On 30 December 2022 19:20:42 GMT, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > >>> Hi Major, > >>> > >>> Thanks for the report! > >>> > >>> On Wed, Dec 21, 2022 at 08:38:46PM +0530, Major Saheb wrote: > >>>> I have an ubuntu guest running on kvm , and I am passing it 10 qemu > >>>> emulated nvme drives > >>>> <iommu model='intel'> > >>>> <driver intremap='on' eim='on'/> > >>>> </iommu> > >>>> <qemu:arg value='pcie-root-port,id=pcie-root-port%d,slot=%d'/> > >>>> <qemu:arg value='nvme,drive=NVME%d,serial=%s_%d,id=NVME%d,bus=pcie-root-port%d'/> > >>>> > >>>> kernel > >>>> Linux node-1 5.15.0-56-generic #62-Ubuntu SMP ----- x86_64 x86_64 > >>>> x86_64 GNU/Linux > >>>> > >>>> kernel command line > >>>> intel_iommu=on > >>>> > >>>> I have attached these drives to vfio-pcie. > >>>> > >>>> when I try to send IO commands to these drives VIA a userspace nvme > >>>> driver using VFIO I get > >>>> [ 1474.752590] DMAR: DRHD: handling fault status reg 2 > >>>> [ 1474.754463] DMAR: [DMA Read NO_PASID] Request device [0b:00.0] > >>>> fault addr 0xffffe000 [fault reason 0x06] PTE Read access is not set > >>>> > >>>> Can someone explain to me what's happening here ? > > > >You can enable iommu debugfs (CONFIG_INTEL_IOMMU_DEBUGFS=y) to check > >the mapping. In this file, you can see if the 0xffffe000 is mapped or > >not. > > > >/sys/kernel/debug/iommu/intel/domain_translation_struct > > My first guess would be that it *was* using queues mapped at that address, but was taken out of the IOMMU domain to be given to userspace, without stopping them.
