The series grew from Eric's idea and patch at [0]. The idea of using the
shortcut for direct DMA as well belongs to Chris.
When an architecture doesn't need DMA synchronization and the buffer is
not an SWIOTLB buffer, most of times the kernel and the drivers end up
calling DMA sync operations for nothing.
Even when DMA is direct, this involves a good non-inline call ladder and
eats a bunch of CPU time. With IOMMU, this results in calling indirect
calls on hotpath just to check what is already known and return.
XSk is been using a custom shortcut for that for quite some time.
I recently wanted to introduce a similar one for Page Pool. Let's combine
all this into one generic shortcut, which would cover all DMA sync ops
and all types of DMA (direct, IOMMU, ...).
* #1 adds stub inlines to be able to skip DMA sync ops or even compile
them out when not needed.
* #2 adds the generic shortcut and enables it for direct DMA.
* #3 adds ability to skip DMA syncs behind an IOMMU.
* #4-5 are just cleanups for Page Pool to avoid merge conflicts in future.
* #6 checks for the shortcut as early as possible in the Page Pool code to
make sure no cycles wasted.
* #7 replaces XSk's shortcut with the generic one.
On 100G NIC, the result is +3-5% for direct DMA and +10-11% for IOMMU.
As a bonus, XSk core now allows batched buffer allocations for IOMMU
setups.
If the shortcut is not available on some system, there should be no
visible performance regressions.
[0] https://lore.kernel.org/netdev/20221115182841.2640176-1-edumazet@xxxxxxxxxx
Alexander Lobakin (5):
dma: compile-out DMA sync op calls when not used
page_pool: make sure frag API fields don't span between cachelines
page_pool: don't use driver-set flags field directly
page_pool: check for DMA sync shortcut earlier
xsk: use generic DMA sync shortcut instead of a custom one
Eric Dumazet (2):
dma: avoid expensive redundant calls for sync operations
iommu/dma: avoid expensive indirect calls for sync operations
kernel/dma/Kconfig | 4 +
include/net/page_pool/types.h | 21 +++-
include/linux/device.h | 5 +
include/linux/dma-map-ops.h | 17 +++
include/linux/dma-mapping.h | 100 +++++++++++++-----
include/net/xdp_sock_drv.h | 7 +-
include/net/xsk_buff_pool.h | 13 +--
drivers/base/dd.c | 2 +
drivers/iommu/dma-iommu.c | 1 +
drivers/net/ethernet/engleder/tsnep_main.c | 2 +-
.../net/ethernet/freescale/dpaa2/dpaa2-xsk.c | 2 +-
drivers/net/ethernet/intel/i40e/i40e_xsk.c | 2 +-
drivers/net/ethernet/intel/ice/ice_xsk.c | 2 +-
drivers/net/ethernet/intel/igc/igc_main.c | 2 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 +-
.../ethernet/mellanox/mlx5/core/en/xsk/rx.c | 4 +-
.../net/ethernet/mellanox/mlx5/core/en_rx.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfd3/xsk.c | 2 +-
.../net/ethernet/stmicro/stmmac/stmmac_main.c | 2 +-
kernel/dma/mapping.c | 60 ++++++++---
kernel/dma/swiotlb.c | 14 +++
net/core/page_pool.c | 67 +++++++-----
net/xdp/xsk_buff_pool.c | 29 +----
23 files changed, 237 insertions(+), 125 deletions(-)
--
2.43.0
