History:
NVIDIA’s Xavier (Tegra194) SOC has multiple SMMU instances that must be
coordinated together. Specifically, there are two instances of ARM’s
MMU-500 shared between coherent DMA devices and one instance of ARM
MMU-500 for non-coherent DMA devices. The two MMU-500s were created to
double the memory bandwidth of a single MMU-500. This is the reason why
Tegra194 does not work with anything but our own Linux fork.
An IOMMU’able device will not know which SMMU it will use. For example,
a device’s memory request could go to MMU 1 or MMU 2. The coherent
memory request gets swizzled in hardware between the two depending on
bandwidth.
For this reason we program the two coherent MMUs identically. They share
page table memory so a device will get the same physical address if it
queries MMU 1 or MMU 2.
Our current implementation involves overriding the write_l and write_q
functions. The new functions write to each of the 3 SMMU registers by
getting the offset from the first SMMU’s base register. We understand
that this is not a permanent fix.
Proposal - Updating arm-smmu.c driver:
Extend the SMMUv2 device tree binding to allow a single device tree node
to represent N instances of an SMMUv2. The cleanest way to do this would
be to allow subnodes for the N-1 instances (For example with 3 SMMUs we
would have the main node and 2 sub nodes). The subnodes would have a
“reg” parameter for the address range and the smmu’s interrupt lines.
We would also need a way for devices to specify which SMMUs they would
like to attach to.
We believe this proposal will have the best performance. Because we are
working in the arm-smmu.c file, we can easily have the multiple SMMUs
share page tables and we can do all of the TLB flushing in parallel.
Please let us know if this proposal will work, or if there is a better
solution. We will create and test patches once we know that we will
have the support of upstream.
Best,
David
