Hi Jean, > -----Original Message----- > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@xxxxxxxxxxxxxxxxxxx] > On Behalf Of Jean-Philippe Brucker > Sent: 02 June 2020 10:39 > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@xxxxxxxxxx> > Cc: devicetree@xxxxxxxxxxxxxxx; kevin.tian@xxxxxxxxx; will@xxxxxxxxxx; > fenghua.yu@xxxxxxxxx; jgg@xxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; > felix.kuehling@xxxxxxx; hch@xxxxxxxxxxxxx; linux-mm@xxxxxxxxx; > iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; catalin.marinas@xxxxxxx; > zhangfei.gao@xxxxxxxxxx; robin.murphy@xxxxxxx; > christian.koenig@xxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > platform devices > > Hi Shameer, > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > wrote: > > > /* IRQ and event handlers */ > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > +*evt) { > > > + int ret; > > > + u32 perm = 0; > > > + struct arm_smmu_master *master; > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > + struct iommu_fault_event fault_evt = { }; > > > + struct iommu_fault *flt = &fault_evt.fault; > > > + > > > + /* Stage-2 is always pinned at the moment */ > > > + if (evt[1] & EVTQ_1_S2) > > > + return -EFAULT; > > > + > > > + master = arm_smmu_find_master(smmu, sid); > > > + if (!master) > > > + return -EINVAL; > > > + > > > + if (evt[1] & EVTQ_1_READ) > > > + perm |= IOMMU_FAULT_PERM_READ; > > > + else > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > + > > > + if (evt[1] & EVTQ_1_EXEC) > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > + > > > + if (evt[1] & EVTQ_1_PRIV) > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > + > > > + if (evt[1] & EVTQ_1_STALL) { > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > + flt->prm = (struct iommu_fault_page_request) { > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > + .perm = perm, > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > + }; > > > + > > > > > + if (ssid_valid) > > > + flt->prm.flags |= > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > Do we need to set this for STALL mode only support? I had an issue > > with this being set on a vSVA POC based on our D06 zip device(which is > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > and stag only. > > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall > event and RESUME command in your prototype? Are you getting issues with > the host driver or the guest driver? The issue is on the host side iommu_page_response(). The flow is something like below. Stall: Host:- arm_smmu_handle_evt() iommu_report_device_fault() vfio_pci_iommu_dev_fault_handler() Stall: Qemu:- vfio_dma_fault_notifier_handler() inject_faults() smmuv3_inject_faults() Stall: Guest:- arm_smmu_handle_evt() iommu_report_device_fault() iommu_queue_iopf ... iopf_handle_group() iopf_handle_single() handle_mm_fault() iopf_complete() iommu_page_response() arm_smmu_page_response() arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) Resume: Qemu:- smmuv3_cmdq_consume(SMMU_CMD_RESUME) smmuv3_notify_page_resp() vfio:ioctl(page_response) --> struct iommu_page_response is filled with only version, grpid and code. Resume: Host:- ioctl(page_response) iommu_page_response() --> fails as the pending req has PASID_VALID flag set and it checks for a match. arm_smmu_page_response() Hope the above is clear. > We do need to forward the SSV flag all the way to the guest driver, so the guest > can find the faulting address space using the SSID. Once the guest handled the > fault, then we don't send the SSID back to the host as part of the RESUME > command. True, the guest requires SSV flag to handle the page fault. But, as shown in the flow above, the issue is on the host side iommu_page_response() where it searches for a matching pending req based on pasid. Not sure we can bypass that and call arm_smmu_page_response() directly but then have to delete the pending req from the list as well. Please let me know if there is a better way to handle the host side page response. Thanks, Shameer > Thanks, > Jean > > > Hence, it is difficult for > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > can see that this flag is being checked in iopf_handle_single() (patch > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let > me know your thoughts. > > > > Thanks, > > Shameer > > > > 1. > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > 5897e4becfa378d15 > > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
