Hi All,
This patch series converts s390's PCI support from its platform specific DMA
API implementation in arch/s390/pci/pci_dma.c to the common DMA IOMMU layer.
The conversion itself is done in patches 3-4 with patch 2 providing the final
necessary IOMMU driver improvement to handle s390's special IOTLB flush
out-of-resource indication in virtualized environments. The conversion
itself only touches the s390 IOMMU driver and s390 arch code moving over
remaining functions from the s390 DMA API implementation. No changes to
common code are necessary.
After patch 4 the basic conversion is done and on our partitioning
machine hypervisor LPAR performance matches the previous implementation.
When running under z/VM or KVM however, performance plummets to about
half of the existing code due to a much higher rate of IOTLB flushes for
unmapped pages. Due to the hypervisors use of IOTLB flushes to
synchronize their shadow tables these are very expensive and minimizing
them is key for regaining the performance loss.
To this end patches 5-6 add a new, single queue, IOTLB flushing scheme
as an alternative to the existing per-CPU flush queues. Introducing an
alternative scheme was suggested by Robin Murphy[1]. The single queue
mode is introduced in patch 4 together with a new .shadow_on_flush flag
bit in struct dev_iommu. This allows IOMMU drivers to indicate that
their IOTLB flushes do the extra work of shadowing. This then lets the
dma-iommu code use a single queue.
Then patch 6 enables variable queue sizes using power of 2 values and
shift/mask to keep performance as close to the fixed size queue code as
possible. A larger queue size and timeout is used by dma-iommu when
shadow_on_flush is set. This same scheme may also be used by other IOMMU
drivers with similar requirements. Particularly virtio-iommu may be
a candidate.
I tested this code on s390x with LPAR, z/VM and KVM hypervisors on an
AMD Ryzen x86 system with native IOMMU and a guest with a modified
virtio-iommu[4] that set .shadow_on_flush = true.
This code is also available in the b4/dma_iommu topic branch of my
git.kernel.org repository[3] with tags matching the version sent.
NOTE: Due to the large drop in performance I think we should not merge
the DMA API conversion (patch 4) until we have a more suited IOVA
flushing scheme with similar improvements as the proposed changes.
NOTE: With the DMA API converted mlx5_core is broken on s390x since commit
06cd555f73ca ("net/mlx5: split mlx5_cmd_init() to probe and reload
routines"). A fix for the breakage has been proposed[5].
Best regards,
Niklas
[0] https://lore.kernel.org/linux-iommu/20221109142903.4080275-1-schnelle@xxxxxxxxxxxxx/
[1] https://lore.kernel.org/linux-iommu/3e402947-61f9-b7e8-1414-fde006257b6f@xxxxxxx/
[2] https://lore.kernel.org/linux-iommu/a8e778da-7b41-a6ba-83c3-c366a426c3da@xxxxxxx/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/
[4] https://lore.kernel.org/lkml/20230726111433.1105665-1-schnelle@xxxxxxxxxxxxx/
[5] https://lore.kernel.org/netdev/20230928-mlx5_init_fix-v1-1-79749d45ce60@xxxxxxxxxxxxx/
---
Changes in v13:
- Rebase on iommu/core which requires removing the platform domain
handling with.
- Moved setting iommu_dma_forcedac = true to s390_iommu_init() so it
happens before the call to iommu_dma_set_pci_32bit_workaround()
- Link to v12: https://lore.kernel.org/r/20230825-dma_iommu-v12-0-4134455994a7@xxxxxxxxxxxxx
Changes in v12:
- Rebased on v6.5-rc7
- Changed queue type flag to an enum
- Incorporated feedback from Robin Murphy
- Set options centrally and only once in iommu_dma_init_domain() with
new helper iommu_dma_init_options()
- Do not reset options of failing to init FQ
- Fixed rebase mishap that partially rolled back patch 2
- Simplified patch 4 by simply no claiming the deferred flush
capability for ISM
- Inlined and removed fq_flush_percpu()
- Changed vzalloc() to vmalloc() for queue
- Added Acked-by's from Robin Murphy
- Link to v11: https://lore.kernel.org/r/20230717-dma_iommu-v11-0-a7a0b83c355c@xxxxxxxxxxxxx
Changes in v11:
- Rebased on v6.5-rc2
- Added patch to force IOMMU_DOMAIN_DMA on s390 specific ISM devices
- Dropped the patch to properly set DMA mask on ISM devices which went upstream separately.
- s390 IOMMU driver now uses IOMMU_CAP_DEFERRED_FLUSH to enable DMA-FQ
leaving no uses of IOMMU_DOMAIN_DMA_FQ in the driver.
- Link to v10: https://lore.kernel.org/r/20230310-dma_iommu-v10-0-f1fbd8310854@xxxxxxxxxxxxx
Changes in v10:
- Rebased on v6.4-rc3
- Removed the .tune_dma_iommu() op in favor of a .shadow_on_flush flag
in struct dev_iommu which then let's the dma-iommu choose a single
queue and larger timeouts and IOVA counts. This leaves the dma-iommu
with full responsibility for the settings.
- The above change affects patches 5 and 6 and lead to a new subject for
patch 6 since the flush queue size and timeout is no longer driver
controlled
- Link to v9: https://lore.kernel.org/r/20230310-dma_iommu-v9-0-65bb8edd2beb@xxxxxxxxxxxxx
Changes in v9:
- Rebased on v6.4-rc2
- Re-ordered iommu_group_store_type() to allow passing the device to
iommu_dma_init_fq()
- Link to v8: https://lore.kernel.org/r/20230310-dma_iommu-v8-0-2347dfbed7af@xxxxxxxxxxxxx
---
Niklas Schnelle (6):
iommu: Allow .iotlb_sync_map to fail and handle s390's -ENOMEM return
s390/pci: prepare is_passed_through() for dma-iommu
s390/pci: Use dma-iommu layer
iommu/s390: Disable deferred flush for ISM devices
iommu/dma: Allow a single FQ in addition to per-CPU FQs
iommu/dma: Use a large flush queue and timeout for shadow_on_flush
Documentation/admin-guide/kernel-parameters.txt | 9 +-
arch/s390/include/asm/pci.h | 7 -
arch/s390/include/asm/pci_clp.h | 3 +
arch/s390/include/asm/pci_dma.h | 119 +---
arch/s390/pci/Makefile | 2 +-
arch/s390/pci/pci.c | 22 +-
arch/s390/pci/pci_bus.c | 5 -
arch/s390/pci/pci_debug.c | 12 +-
arch/s390/pci/pci_dma.c | 735 ------------------------
arch/s390/pci/pci_event.c | 17 +-
arch/s390/pci/pci_sysfs.c | 19 +-
drivers/iommu/Kconfig | 4 +-
drivers/iommu/amd/iommu.c | 5 +-
drivers/iommu/apple-dart.c | 5 +-
drivers/iommu/dma-iommu.c | 200 +++++--
drivers/iommu/intel/iommu.c | 5 +-
drivers/iommu/iommu.c | 20 +-
drivers/iommu/msm_iommu.c | 5 +-
drivers/iommu/mtk_iommu.c | 5 +-
drivers/iommu/s390-iommu.c | 434 ++++++++++++--
drivers/iommu/sprd-iommu.c | 5 +-
drivers/iommu/sun50i-iommu.c | 6 +-
include/linux/iommu.h | 6 +-
23 files changed, 634 insertions(+), 1016 deletions(-)
---
base-commit: ccb76c5751634541bfd2222374f46e1dea83d525
change-id: 20230310-dma_iommu-5e048c538647
Best regards,
--
Niklas Schnelle
Linux on Z Development
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement - https://www.ibm.com/privacy
