Hello, I haven't seen any reply to this e-mail. I know that the combination of IOMMU, DMA mapping and DT doesn't exactly sound like fun, but I think we still need to move on :-) Will and Arnd, could you please confirm that my summary below matches your memories of our discussion ? On Monday 16 February 2015 18:14:45 Laurent Pinchart wrote: > On Thursday 05 February 2015 16:31:58 Laura Abbott wrote: > > Hi, > > > > On the heels of Exynos integrating SMMU in to the DT for probing, > > this series attempts to add support to make SMMU drivers work with > > deferred probing. This has been referenced before[1] but this is > > some actual code to use as a starting point for discussion. > > > > The requirement for this is based on a previous patch to add clock > > support to the ARM SMMU driver[2]. Once we have clock support, it's > > possible that the driver itself may need to be defered which breaks > > a bunch of assumptions about how SMMU probing is supposed to work. > > The concept here is to have the driver call of_dma_configure which > > may return -EPROBE_DEFER. of_dma_configure could possibly be moved > > up to probe level. The existing method of initialization still needs > > to be done as well which might mean we have the worst of both worlds. > > > > Open questions: > > - This still doesn't really address Arnd's concerns[3] about disabling > > IOMMUs > > Arnd, Will and I have discussed IOMMU probe deferral last week. Here's a > summary of the discussion, before the details blur away. > > The discussion covered both higher level concepts and lower level details, > in various directions. I'll try to make the summary clear by filling the > gaps between pieces where needed, so some of the information below might > not be a direct result of the discussions. Arnd and Will, please feel free > to correct me. > > The first question we've discussed was whether probe deferral for IOMMUs is > really needed. Various dependencies between IOMMU devices and other devices > exist, in particular on clocks (as you have mentioned above) and on power > domains (as mentioned by Marek in > http://lists.infradead.org/pipermail/linux-arm-kernel/2015-February/323238. > html). While there are mechanism to handle some of them with probe deferral > (for instance by using the OF_DECLARE macros to register clock drivers), > generalizing those mechanisms would essentially recreate a probe ordering > mechanism similar to link order probe ordering and couldn't really scale. > > Additionally, IOMMUs could also be present hot-pluggable devices and depend > on resources that are thus hot-plugged. OF_DECLARE wouldn't help in that > case. For all those reasons we have concluded that probe deferral for > IOMMUs is desired if it can be implemented cleanly. > > The second question we've discussed was how to implement probe deferral > cleanly :-) > > The current implementation configures DMA at device creation time and sets > the following properties: > > - dma_map_ops (IOMMU, SWIOTLB, linear mapping) > - initial DMA mask > - DMA offset > - DMA coherency > > Additionally the IOMMU group (when an IOMMU is present) will also be > configured at the same time (patches are under review). > > The base idea is to defer probing of bus master devices when their IOMMU > isn't present. Three cases need to be handled. > > 1. No IOMMU is declared by the firmware (through DT, ACPI, ...) for the bus > master device. The bus master device probe doesn't need to be deferred due > to the IOMMU. dma_map_ops must be set to SWIOTLB or linear mapping (the > later should likely be implemented through SWIOTLB). > > 2. An IOMMU is declared for the bus master device and the IOMMU has been > successfully probed and registered. The bus master device probe doesn't need > to be deferred due to the IOMMU. dma_map_ops must be set to IOMMU ops. > > 3. An IOMMU is declared for the bus master device but the IOMMU isn't > registered. This can be caused by different reasons: > > - a. No driver is loaded for this IOMMU (for instance because DT describes > the IOMMU connection, but the IOMMU driver hasn't been developed yet, or an > older kernel is used). If the IOMMU is optional the bus master device probe > should succeed, and dma_map_ops should be set to linear. If the IOMMU is > mandatory the bus master device probe should fail. > > Note that, as we require IOMMU drivers to be compiled in, we don't need to > handle the case of loadable IOMMU drivers that haven't been loaded yet. > > - b. A driver is loaded for this IOMMU but the IOMMU hasn't been probed yet, > or its probe has been deferred. The bus master device probe should be > deferred. > > - c. A driver is loaded for this IOMMU but the IOMMU probe has failed > permanently. It's not clear at the moment whether we should try to recover > from this automatically using the same mechanism as case 3.a, or if we can > considered this as an abnormal failure and disable the bus master device. > > Assuming that we should try to recover from such an error, we can't predict > this case when the device is instantiated (even if IOMMUs are registered > before bus master devices are added, for instance using the OF_DECLARE > mechanism that Will has implemented). We thus need to setup the dma_map_ops > and IOMMU group at bus master device probe time. > > Furthermore, we can't distinguish cases 3.a and 3.b at bus master probe time > without early registration of IOMMU drivers. Case 3.a would instead be > considered as 3.b, leading to infinite probe deferral of bus master > devices. > > For those reasons we have concluded that IOMMU probe deferral needs to be > implemented with a combination of several mechanisms. The following steps > should happen at bus master device probe time. > > 1. The IOMMU device referenced by the firmware with the bus master device is > looked up. On DT-based systems, this will be the IOMMU DT node referenced > by the iommus property. If no IOMMU device is associated, dma_map_ops will > be set to linear mapping or SWIOTLB and device probe will continue. > > 2. An IOMMU device is referenced for the bus master device. > > The corresponding IOMMU instance is looked up. This requires a new IOMMU > registration system. IOMMU drivers will create IOMMU instances at probe time > and register them with the IOMMU core. > > If an IOMMU instance is found for the referenced IOMMU device, the IOMMU > instance's of_xlate function will be called to setup the IOMMU. > > If the of_xlate call succeeds dma_map_ops will be set to IOMMU ops and > device probe will continue. If the call fails we can either fail the bus > master device probe, or fall back to non-IOMMU dma_map_ops (to be > discussed). > > 3. The IOMMU device referenced for the bus master device isn't present, due > to the IOMMU device probe not having been performed yet, having been > deferred, or having failed. > > The IOMMU driver associated with the IOMMU device is looked up. This was > initially thought to require an early registration mechanism for IOMMU > drivers (using an OF_DECLARE mechanism for DT-based systems for instance), > but on second thought it might be possible to implement this based on > normal driver registration (to be researched). > > If an IOMMU driver is found for the referenced IOMMU device, a callback > function of the IOMMU driver is called to check whether an IOMMU instance is > expected to be registered later (most IOMMU drivers will just return true, > so we could skip this callback function until an IOMMU driver requires it). > If an IOMMU instance is expected to be registered later the bus master > device probe is deferred. Otherwise dma_map_ops will be set to linear > mapping/SWIOTLB and device probe will continue. > > > The initial DMA mask and the DMA offset can still be configured at device > instantiation time if desired. > > > - Currently tested where we knew the driver was going to be deferring. > > Probably need some logic for calling of_dma_configure again. > > > > This is based on Robin Murphy's work for dma mapping[4] and a few > > patches from Murali Kaicheri[5] for of_dma_configure. > > > > > > [1] http://lists.linuxfoundation.org/pipermail/iommu/2015-> > January/011764.html > > [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2014-> > August/279036.html > > [3] http://lists.infradead.org/pipermail/linux-arm-kernel/2014-> > December/311579.html > > [4] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-> > January/315459.html > > [5] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-> > January/319390.html > > > > Laura Abbott (3): > > dma-mapping: Make arch_setup_dma_ops return an error code > > of: Return error codes from of_dma_configure > > iommu/arm-smmu: Support deferred probing > > > > Mitchel Humpherys (1): > > iommu/arm-smmu: add support for specifying clocks > > > > arch/arm/include/asm/dma-mapping.h | 2 +- > > arch/arm/mm/dma-mapping.c | 4 +- > > arch/arm64/include/asm/dma-mapping.h | 2 +- > > arch/arm64/mm/dma-mapping.c | 16 +-- > > drivers/iommu/arm-smmu.c | 186 ++++++++++++++++++++++++++++-- > > drivers/iommu/iommu.c | 49 ++++++++- > > drivers/iommu/of_iommu.c | 14 ++- > > drivers/of/device.c | 9 +- > > include/linux/dma-mapping.h | 7 +- > > include/linux/iommu.h | 2 + > > include/linux/of_device.h | 4 +- > > 11 files changed, 268 insertions(+), 27 deletions(-) -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
