From: Arnd Bergmann <arnd@xxxxxxxx>
After a long discussion about adding SoC specific semantics for when
to flush caches in drivers/soc/ drivers that we determined to be
fundamentally flawed[1], I volunteered to try to move that logic into
architecture-independent code and make all existing architectures do
the same thing.
As we had determined earlier, the behavior is wildly different across
architectures, but most of the differences come down to either bugs
(when required flushes are missing) or extra flushes that are harmless
but might hurt performance.
I finally found the time to come up with an implementation of this, which
starts by replacing every outlier with one of the three common options:
1. architectures without speculative prefetching (hegagon, m68k,
openrisc, sh, sparc, and certain armv4 and xtensa implementations)
only flush their caches before a DMA, by cleaning write-back caches
(if any) before a DMA to the device, and by invalidating the caches
before a DMA from a device
2. arc, microblaze, mips, nios2, sh and later xtensa now follow the
normal 32-bit arm model and invalidate their writeback caches
again after a DMA from the device, to remove stale cache lines
that got prefetched during the DMA. arc, csky and mips used to
invalidate buffers also before the bidirectional DMA, but this
is now skipped whenever we know it gets invalidated again
after the DMA.
3. parisc, powerpc and riscv already flushed buffers before
a DMA_FROM_DEVICE, and these get moved to the arm64 behavior
that does the writeback before and invalidate after both
DMA_FROM_DEVICE and DMA_BIDIRECTIONAL in order to avoid the
problem of accidentally leaking stale data if the DMA does
not actually happen[2].
The last patch in the series replaces the architecture specific code
with a shared version that implements all three based on architecture
specific parameters that are almost always determined at compile time.
The difference between cases 1. and 2. is hardware specific, while between
2. and 3. we need to decide which semantics we want, but I explicitly
avoid this question in my series and leave it to be decided later.
Another difference that I do not address here is what cache invalidation
does for partical cache lines. On arm32, arm64 and powerpc, a partial
cache line always gets written back before invalidation in order to
ensure that data before or after the buffer is not discarded. On all
other architectures, the assumption is cache lines are never shared
between DMA buffer and data that is accessed by the CPU. If we end up
always writing back dirty cache lines before a DMA (option 3 above),
then this point becomes moot, otherwise we should probably address this
in a follow-up series to document one behavior or the other and implement
it consistently.
Please review!
Arnd
[1] https://lore.kernel.org/all/20221212115505.36770-1-prabhakar.mahadev-lad.rj@xxxxxxxxxxxxxx/
[2] https://lore.kernel.org/all/20220606152150.GA31568@willie-the-truck/
Arnd Bergmann (21):
openrisc: dma-mapping: flush bidirectional mappings
xtensa: dma-mapping: use normal cache invalidation rules
sparc32: flush caches in dma_sync_*for_device
microblaze: dma-mapping: skip extra DMA flushes
powerpc: dma-mapping: split out cache operation logic
powerpc: dma-mapping: minimize for_cpu flushing
powerpc: dma-mapping: always clean cache in _for_device() op
riscv: dma-mapping: only invalidate after DMA, not flush
riscv: dma-mapping: skip invalidation before bidirectional DMA
csky: dma-mapping: skip invalidating before DMA from device
mips: dma-mapping: skip invalidating before bidirectional DMA
mips: dma-mapping: split out cache operation logic
arc: dma-mapping: skip invalidating before bidirectional DMA
parisc: dma-mapping: use regular flush/invalidate ops
ARM: dma-mapping: always invalidate WT caches before DMA
ARM: dma-mapping: bring back dmac_{clean,inv}_range
ARM: dma-mapping: use arch_sync_dma_for_{device,cpu}() internally
ARM: drop SMP support for ARM11MPCore
ARM: dma-mapping: use generic form of arch_sync_dma_* helpers
ARM: dma-mapping: split out arch_dma_mark_clean() helper
dma-mapping: replace custom code with generic implementation
arch/arc/mm/dma.c | 66 ++------
arch/arm/Kconfig | 4 +
arch/arm/include/asm/cacheflush.h | 21 +++
arch/arm/include/asm/glue-cache.h | 4 +
arch/arm/mach-oxnas/Kconfig | 4 -
arch/arm/mach-oxnas/Makefile | 1 -
arch/arm/mach-oxnas/headsmp.S | 23 ---
arch/arm/mach-oxnas/platsmp.c | 96 -----------
arch/arm/mach-versatile/platsmp-realview.c | 4 -
arch/arm/mm/Kconfig | 19 ---
arch/arm/mm/cache-fa.S | 4 +-
arch/arm/mm/cache-nop.S | 6 +
arch/arm/mm/cache-v4.S | 13 +-
arch/arm/mm/cache-v4wb.S | 4 +-
arch/arm/mm/cache-v4wt.S | 22 ++-
arch/arm/mm/cache-v6.S | 35 +---
arch/arm/mm/cache-v7.S | 6 +-
arch/arm/mm/cache-v7m.S | 4 +-
arch/arm/mm/dma-mapping-nommu.c | 36 ++--
arch/arm/mm/dma-mapping.c | 181 ++++++++++-----------
arch/arm/mm/proc-arm1020.S | 4 +-
arch/arm/mm/proc-arm1020e.S | 4 +-
arch/arm/mm/proc-arm1022.S | 4 +-
arch/arm/mm/proc-arm1026.S | 4 +-
arch/arm/mm/proc-arm920.S | 4 +-
arch/arm/mm/proc-arm922.S | 4 +-
arch/arm/mm/proc-arm925.S | 4 +-
arch/arm/mm/proc-arm926.S | 4 +-
arch/arm/mm/proc-arm940.S | 4 +-
arch/arm/mm/proc-arm946.S | 4 +-
arch/arm/mm/proc-feroceon.S | 8 +-
arch/arm/mm/proc-macros.S | 2 +
arch/arm/mm/proc-mohawk.S | 4 +-
arch/arm/mm/proc-xsc3.S | 4 +-
arch/arm/mm/proc-xscale.S | 6 +-
arch/arm64/mm/dma-mapping.c | 28 ++--
arch/csky/mm/dma-mapping.c | 46 +++---
arch/hexagon/kernel/dma.c | 44 ++---
arch/m68k/kernel/dma.c | 43 +++--
arch/microblaze/kernel/dma.c | 38 ++---
arch/mips/mm/dma-noncoherent.c | 75 +++------
arch/nios2/mm/dma-mapping.c | 57 +++----
arch/openrisc/kernel/dma.c | 62 ++++---
arch/parisc/include/asm/cacheflush.h | 6 +-
arch/parisc/kernel/pci-dma.c | 33 +++-
arch/powerpc/mm/dma-noncoherent.c | 76 +++++----
arch/riscv/mm/dma-noncoherent.c | 51 +++---
arch/sh/kernel/dma-coherent.c | 43 +++--
arch/sparc/Kconfig | 2 +-
arch/sparc/kernel/ioport.c | 38 +++--
arch/xtensa/Kconfig | 1 -
arch/xtensa/include/asm/cacheflush.h | 6 +-
arch/xtensa/kernel/pci-dma.c | 47 +++---
include/linux/dma-sync.h | 107 ++++++++++++
54 files changed, 721 insertions(+), 699 deletions(-)
delete mode 100644 arch/arm/mach-oxnas/headsmp.S
delete mode 100644 arch/arm/mach-oxnas/platsmp.c
create mode 100644 include/linux/dma-sync.h
--
2.39.2
Cc: Vineet Gupta <vgupta@xxxxxxxxxx>
Cc: Russell King <linux@xxxxxxxxxxxxxxx>
Cc: Neil Armstrong <neil.armstrong@xxxxxxxxxx>
Cc: Linus Walleij <linus.walleij@xxxxxxxxxx>
Cc: Catalin Marinas <catalin.marinas@xxxxxxx>
Cc: Will Deacon <will@xxxxxxxxxx>
Cc: Guo Ren <guoren@xxxxxxxxxx>
Cc: Brian Cain <bcain@xxxxxxxxxxx>
Cc: Geert Uytterhoeven <geert@xxxxxxxxxxxxxx>
Cc: Michal Simek <monstr@xxxxxxxxx>
Cc: Thomas Bogendoerfer <tsbogend@xxxxxxxxxxxxxxxx>
Cc: Dinh Nguyen <dinguyen@xxxxxxxxxx>
Cc: Stafford Horne <shorne@xxxxxxxxx>
Cc: Helge Deller <deller@xxxxxx>
Cc: Michael Ellerman <mpe@xxxxxxxxxxxxxx>
Cc: Christophe Leroy <christophe.leroy@xxxxxxxxxx>
Cc: Paul Walmsley <paul.walmsley@xxxxxxxxxx>
Cc: Palmer Dabbelt <palmer@xxxxxxxxxxx>
Cc: Rich Felker <dalias@xxxxxxxx>
Cc: John Paul Adrian Glaubitz <glaubitz@xxxxxxxxxxxxxxxxxxx>
Cc: "David S. Miller" <davem@xxxxxxxxxxxxx>
Cc: Max Filippov <jcmvbkbc@xxxxxxxxx>
Cc: Christoph Hellwig <hch@xxxxxx>
Cc: Robin Murphy <robin.murphy@xxxxxxx>
Cc: Lad Prabhakar <prabhakar.mahadev-lad.rj@xxxxxxxxxxxxxx>
Cc: Conor Dooley <conor.dooley@xxxxxxxxxxxxx>
Cc: linux-snps-arc@xxxxxxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
Cc: linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
Cc: linux-oxnas@xxxxxxxxx
Cc: linux-csky@xxxxxxxxxxxxxxx
Cc: linux-hexagon@xxxxxxxxxxxxxxx
Cc: linux-m68k@xxxxxxxxxxxxxxxxxxxx
Cc: linux-mips@xxxxxxxxxxxxxxx
Cc: linux-openrisc@xxxxxxxxxxxxxxx
Cc: linux-parisc@xxxxxxxxxxxxxxx
Cc: linuxppc-dev@xxxxxxxxxxxxxxxx
Cc: linux-riscv@xxxxxxxxxxxxxxxxxxx
Cc: linux-sh@xxxxxxxxxxxxxxx
Cc: sparclinux@xxxxxxxxxxxxxxx
Cc: linux-xtensa@xxxxxxxxxxxxxxxx
