On 2024-04-30 1:41 am, Dmitry Baryshkov wrote:
On Tue, 30 Apr 2024 at 01:26, Robin Murphy <robin.murphy@xxxxxxx> wrote:
On 2024-04-29 5:31 pm, Dmitry Baryshkov wrote:
On Fri, Apr 19, 2024 at 05:54:45PM +0100, Robin Murphy wrote:
It's somewhat hard to see, but arm64's arch_setup_dma_ops() should only
ever call iommu_setup_dma_ops() after a successful iommu_probe_device(),
which means there should be no harm in achieving the same order of
operations by running it off the back of iommu_probe_device() itself.
This then puts it in line with the x86 and s390 .probe_finalize bodges,
letting us pull it all into the main flow properly. As a bonus this lets
us fold in and de-scope the PCI workaround setup as well.
At this point we can also then pull the call up inside the group mutex,
and avoid having to think about whether iommu_group_store_type() could
theoretically race and free the domain if iommu_setup_dma_ops() ran just
*before* iommu_device_use_default_domain() claims it... Furthermore we
replace one .probe_finalize call completely, since the only remaining
implementations are now one which only needs to run once for the initial
boot-time probe, and two which themselves render that path unreachable.
This leaves us a big step closer to realistically being able to unpick
the variety of different things that iommu_setup_dma_ops() has been
muddling together, and further streamline iommu-dma into core API flows
in future.
Reviewed-by: Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx> # For Intel IOMMU
Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Tested-by: Hanjun Guo <guohanjun@xxxxxxxxxx>
Signed-off-by: Robin Murphy <robin.murphy@xxxxxxx>
---
v2: Shuffle around to make sure the iommu_group_do_probe_finalize() case
is covered as well, with bonus side-effects as above.
v3: *Really* do that, remembering the other two probe_finalize sites too.
---
arch/arm64/mm/dma-mapping.c | 2 --
drivers/iommu/amd/iommu.c | 8 --------
drivers/iommu/dma-iommu.c | 18 ++++++------------
drivers/iommu/dma-iommu.h | 14 ++++++--------
drivers/iommu/intel/iommu.c | 7 -------
drivers/iommu/iommu.c | 20 +++++++-------------
drivers/iommu/s390-iommu.c | 6 ------
drivers/iommu/virtio-iommu.c | 10 ----------
include/linux/iommu.h | 7 -------
9 files changed, 19 insertions(+), 73 deletions(-)
This patch breaks UFS on Qualcomm SC8180X Primus platform:
[ 3.846856] arm-smmu 15000000.iommu: Unhandled context fault: fsr=0x402, iova=0x1032db3e0, fsynr=0x130000, cbfrsynra=0x300, cb=4
Hmm, a context fault implies that the device did get attached to a DMA
domain, thus has successfully been through __iommu_probe_device(), yet
somehow still didn't get the right DMA ops (since that "IOVA" looks more
like a PA to me). Do you see the "Adding to IOMMU group..." message for
this device, and/or any other relevant messages or errors before this
point?
No, nothing relevant.
[ 8.372395] ufshcd-qcom 1d84000.ufshc: Adding to iommu group 6
(please ignore the timestamp, it comes before ufshc being probed).
I'm guessing there's a fair chance probe deferral might be
involved as well. I'd like to understand what path(s) this ends up
taking through __iommu_probe_device() and of_dma_configure(), or at
least the number and order of probe attempts between the UFS and SMMU
drivers.
__iommu_probe_device() gets called twice and returns early because ops is NULL.
Then finally of_dma_configure_id() is called. The following branches are taken:
np == dev->of_node
of_dma_get_range() returned 0
bus_dma_limit and dma_range_map are set
__iommu_probe_device() is called, using the `!group->default_domain &&
!group_lis` case, then group->default_domain() is not NULL,
In the end, iommu_setup_dma_ops() is called.
Then the ufshc probe defers (most likely the PHY is not present or
some other device is not there yet).
Ah good, probe deferral. And indeed the half-formed hunch from last
night grew into a pretty definite idea by this morning... patch incoming.
Thanks,
Robin.
On the next (succeeding) try, of_dma_configure_id() is called again.
The call trace is more or less the same, except that
__iommu_probe_device() is not called
I'll stare at the code in the morning and see if I can spot any
overlooked ways in which what I think might be happening could happen,
but any more info to help narrow it down would be much appreciated.
Thanks,
Robin.
[ 3.846880] ufshcd-qcom 1d84000.ufshc: ufshcd_check_errors: saved_err 0x20000 saved_uic_err 0x0
[ 3.846929] host_regs: 00000000: 1587031f 00000000 00000300 00000000
[ 3.846935] host_regs: 00000010: 01000000 00010217 00000000 00000000
[ 3.846941] host_regs: 00000020: 00000000 00070ef5 00000000 00000000
[ 3.846946] host_regs: 00000030: 0000000f 00000001 00000000 00000000
[ 3.846951] host_regs: 00000040: 00000000 00000000 00000000 00000000
[ 3.846956] host_regs: 00000050: 032db000 00000001 00000000 00000000
[ 3.846962] host_regs: 00000060: 00000000 80000000 00000000 00000000
[ 3.846967] host_regs: 00000070: 032dd000 00000001 00000000 00000000
[ 3.846972] host_regs: 00000080: 00000000 00000000 00000000 00000000
[ 3.846977] host_regs: 00000090: 00000016 00000000 00000000 0000000c
[ 3.847074] ufshcd-qcom 1d84000.ufshc: ufshcd_err_handler started; HBA state eh_fatal; powered 1; shutting down 0; saved_err = 131072; saved_uic_err = 0; force_reset = 0
[ 4.406550] ufshcd-qcom 1d84000.ufshc: ufshcd_verify_dev_init: NOP OUT failed -11
[ 4.417953] ufshcd-qcom 1d84000.ufshc: ufshcd_async_scan failed: -11
