On Wed, Sep 04, 2024 at 07:40:43PM -0700, Nicolin Chen wrote:
> It's observed that, when the first 4GB of system memory was reserved, all
> VCMDQ allocations failed (even with the smallest qsz in the last attempt):
> arm-smmu-v3: found companion CMDQV device: NVDA200C:00
> arm-smmu-v3: option mask 0x10
> arm-smmu-v3: failed to allocate queue (0x8000 bytes) for vcmdq0
> acpi NVDA200C:00: tegra241_cmdqv: Falling back to standard SMMU CMDQ
> arm-smmu-v3: ias 48-bit, oas 48-bit (features 0x001e1fbf)
> arm-smmu-v3: allocated 524288 entries for cmdq
> arm-smmu-v3: allocated 524288 entries for evtq
> arm-smmu-v3: allocated 524288 entries for priq
>
> This is because the 4GB reserved memory shifted the entire DMA zone from a
> lower 32-bit range (on a system without the 4GB carveout) to higher range,
> while the dev->coherent_dma_mask was set to DMA_BIT_MASK(32) by default.
>
> The dma_set_mask_and_coherent() call is done in arm_smmu_device_hw_probe()
> of the SMMU driver. So any DMA allocation from tegra241_cmdqv_probe() must
> wait until the coherent_dma_mask is correctly set.
>
> Move the vintf/vcmdq structure initialization routine into a different op,
> "init_structures". Call it at the end of arm_smmu_init_structures(), where
> standard SMMU queues get allocated.
>
> Most of the impl_ops aren't ready until vintf/vcmdq structure are init-ed.
> So replace the full impl_ops with an init_ops in __tegra241_cmdqv_probe().
>
> And switch to tegra241_cmdqv_impl_ops later in arm_smmu_init_structures().
> Note that tegra241_cmdqv_impl_ops does not link to the new init_structures
> op after this switch, since there is no point in having it once it's done.
>
> Fixes: 918eb5c856f6 ("iommu/arm-smmu-v3: Add in-kernel support for NVIDIA Tegra241 (Grace) CMDQV")
> Reported-by: Matt Ochs <mochs@xxxxxxxxxx>
> Signed-off-by: Nicolin Chen <nicolinc@xxxxxxxxxx>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 9 +-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 +
> .../iommu/arm/arm-smmu-v3/tegra241-cmdqv.c | 83 ++++++++++++-------
> 3 files changed, 60 insertions(+), 33 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Jason
