On Mon, Dec 9, 2024 at 2:49 AM Robin Murphy <robin.murphy@xxxxxxx> wrote:
>
> On 09/12/2024 8:17 am, Christoph Hellwig wrote:
> > I scrolled three pages before giving up as it was just quotes over
> > quotas. Can you please write an email that contains whatever you're
> > trying to tell instead of just quotes? Same for the person replying.
>
Christoph,
Understood; there was a lot of back and forth and likely some
misinformation from my early replies. Let me recap here.
The issue I run into is that with Linux 6.9 and beyond on an IMX8M
Mini SoC (no IMOMMU) with >3GiB DRAM (which requires more than 32 bits
of address due to IMX8M's DRAM base being at 0x40000000) the ath11k
driver will fail to register a netdev and errors out with 'ath11k
swiotlb buffer is full':
[ 8.057077] ath11k_pci 0000:04:00.0: BAR 0 [mem
0x18200000-0x183fffff 64bit]: assigned
[ 8.057151] ath11k_pci 0000:04:00.0: enabling device (0000 -> 0002)
[ 8.091920] ath11k_pci 0000:04:00.0: MSI vectors: 16
[ 8.091960] ath11k_pci 0000:04:00.0: qcn9074 hw1.0
[ 8.832924] ath11k_pci 0000:04:00.0: chip_id 0x0 chip_family 0x0
board_id 0xff soc_id 0xffffffff
[ 8.832951] ath11k_pci 0000:04:00.0: fw_version 0x270206d0
fw_build_timestamp 2022-08-04 12:48 fw_build_id
WLAN.HK.2.7.0.1-01744-QCAHKSWPL_SILICONZ-1
[ 10.194343] ath11k_pci 0000:04:00.0: swiotlb buffer is full (sz:
1048583 bytes), total 32768 (slots), used 2529 (slots)
[ 10.194406] ath11k_pci 0000:04:00.0: failed to set up tcl_comp ring (0) :-12
[ 10.194781] ath11k_pci 0000:04:00.0: failed to init DP: -12
After a lot of back and forth and investigation this is due to the
IMX8M SoC's not having an IOMMU thus swiotlb is being used and ath11k
is requesting some buffers that are too large for swiotlb to provide.
There is a specific patch which added the HAL_WBM2SW_RELEASE buffers
to cacheable memory that could be reverted to fix this but the concern
was that it would impact performance moving those buffers to
non-cacheable memory (there are three ~1MiB buffers being allocated):
commit d0e2523bfa9cb ("ath11k: allocate HAL_WBM2SW_RELEASE ring from
cacheable memory").
> TBH I'm hesitant to look too closely since everything those Atheros WiFi
> drivers do with DMA tends to be sketchy, but from what I could make out
> from skimming until I also gave up, I think it might be an attempt to
> reinvent dma_alloc_pages(), or possibly dma_alloc_noncoherent().
Robin,
Agreed - I'm not sure how much attention, review, or testing these
ath11k patches originally got due to the fact that there appears to be
breakage for a couple of years here.
The chain of events as best I can tell are:
commit 6452f0a3d565 ("ath11k: allocate dst ring descriptors from
cacheable memory")
- Nov 12 2021 (made it into Linux 5.17)
- changes allocation of reo_dst rings to cacheable memory to allow
cached descriptor access to optimize CPU usage
- this is flawed because it uses virt_to_phys() to allocate cacheable
memory which does not work on systems with an IOMMU enabled or using
software IOMMU (swiotlb); this causes a kernel crash on client
association
commit d0e2523bfa9c ("ath11k: allocate HAL_WBM2SW_RELEASE ring from
cacheable memory)
- Nov 12 2021 (made it into Linux Linux 5.17)
- furthers the previous patch by also including the WBM2SW buffers in
cacheable memory which are about 1MiB in size
commit aaf244141ed7 ("wifi: ath11k: fix IOMMU errors on buffer rings")
- Dec 20 2023 (made it into Linux 6.9)
- resolves the issue from commit 6452f0a3d565 (but missing a Fixes
tag) by changing the virt_to_phys calls to dma_map_single() but on
systems that need software IOMMUC (IMX8MM > 3GiB) this exposes the
'swiotlb buffer is full' limitation due to commit d0e2523bfa9c which
allocates buffers exceeding the 256KiB limit imposed by swiotlb
Therefore in the case of an IOMMU'less system with DMA address
limitations of 32bit and >3GiB DRAM (as many IMX8M boards have) the
ath11k driver has been broken since 5.17.
Best Regards,
Tim
