> Hmm, I did misspeak slightly (it's late and I really should have left
> this for tomorrow...) - that's 12KB per CPU *per domain*, but still that
> would seem to imply well over 100 domains if you have 242MB of magazine
> allocations while the iommu_iova cache isn't even on the charts... what
> the heck is that driver doing?
>
> (I don't necessarily disagree with the spirit of the patch BTW, I just
> really want to understand the situation that prompted it, and make sure
> we don't actually have a subtle leak somewhere.)
Hi Robin,
The following tracing is without Google TPU, simply upstream kernel:
The iova_domain_init_rcaches is called 159 with the following stack:
iova_domain_init_rcaches
iommu_setup_dma_ops
amd_iommu_probe_finalize
bus_iommu_probe
iommu_device_register
iommu_init_pci
amd_iommu_init_pci
state_next
iommu_go_to_state
amd_iommu_init
pci_iommu_init
do_one_initcall
Each time 1536K is allocated: in total 159 * 1536K = 238.5M
The allocation happens like this:
for (IOVA_RANGE_CACHE_MAX_SIZE)
for_each_possible_cpu()
iova_magazine_alloc
iova_magazine_alloc
IOVA_RANGE_CACHE_MAX_SIZE = 6
ncpu = 128
sizeof (struct iova_magazine) = 1K
6 * 128 * (1K + 1K) = 1536K
Pasha
