On Wed, Feb 26, 2025 at 01:07:44PM +0100, Niklas Schnelle wrote:
> With the introduction of memory I/O (MIO) instructions enbaled in commit
> 71ba41c9b1d9 ("s390/pci: provide support for MIO instructions") s390
> gained support for direct user-space access to mapped PCI resources.
> Even without those however user-space can access mapped PCI resources
> via the s390 specific MMIO syscalls. There is thus nothing fundamentally
> preventing s390 from supporting VFIO_PCI_MMAP, allowing user-space
> drivers to access PCI resources without going through the pread()
> interface. To actually enable VFIO_PCI_MMAP a few issues need fixing
> however.
>
> Firstly the s390 MMIO syscalls do not cause a page fault when
> follow_pte() fails due to the page not being present. This breaks
> vfio-pci's mmap() handling which lazily maps on first access.
>
> Secondly on s390 there is a virtual PCI device called ISM which has
> a few oddities. For one it claims to have a 256 TiB PCI BAR (not a typo)
> which leads to any attempt to mmap() it fail with the following message:
>
> vmap allocation for size 281474976714752 failed: use vmalloc=<size> to increase size
>
> Even if one tried to map this BAR only partially the mapping would not
> be usable on systems with MIO support enabled. So just block mapping
> BARs which don't fit between IOREMAP_START and IOREMAP_END. Solve this
> by keeping the vfio-pci mmap() blocking behavior around for this
> specific device via a PCI quirk and new pdev->non_mappable_bars
> flag.
>
> As noted by Alex Williamson With mmap() enabled in vfio-pci it makes
> sense to also enable HAVE_PCI_MMAP with the same restriction for pdev->
> non_mappable_bars. So this is added in patch 3 and I tested this with
> another small test program.
>
> Note:
> For your convenience the code is also available in the tagged
> b4/vfio_pci_mmap branch on my git.kernel.org site below:
> https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/
>
> Thanks,
> Niklas
>
> Link: https://lore.kernel.org/all/c5ba134a1d4f4465b5956027e6a4ea6f6beff969.camel@xxxxxxxxxxxxx/
> Signed-off-by: Niklas Schnelle <schnelle@xxxxxxxxxxxxx>
> ---
> Changes in v7:
> - Move all s390 changes, except for a one-lineer to set pdev->
> non_mappable_bars for all devices, to the third patch (Bjorn)
> - Move checks in pci-sysfs.c and proc.c to the second patch (Bjorn)
> - Only set ARCH_GENERIC_PCI_MMAP_RESOURCES not HAVE_PCI_MMAP following
> the recommendation for new architectures in
> Documentation/PCI/sysfs-pci.rst. This only enables the sysfs but not
> the proc interface.
> - Link to v6: https://lore.kernel.org/r/20250214-vfio_pci_mmap-v6-0-6f300cb63a7e@xxxxxxxxxxxxx
>
> Changes in v6:
> - Add a patch to also enable PCI resource mmap() via sysfs and proc
> exlcluding pdev->non_mappable_bars devices (Alex Williamson)
> - Added Acks
> - Link to v5: https://lore.kernel.org/r/20250212-vfio_pci_mmap-v5-0-633ca5e056da@xxxxxxxxxxxxx
>
> Changes in v5:
> - Instead of relying on the existing pdev->non_compliant_bars introduce
> a new pdev->non_mappable_bars flag. This replaces the VFIO_PCI_MMAP
> Kconfig option and makes it per-device. This is necessary to not break
> upcoming vfio-pci use of ISM devices (Julian Ruess)
> - Squash the removal of VFIO_PCI_MMAP into the second commit as this
> is now where its only use goes away.
> - Switch to using follow_pfnmap_start() in MMIO syscall page fault
> handling to match upstream changes
> - Dropped R-b's because the changes are significant
> - Link to v4: https://lore.kernel.org/r/20240626-vfio_pci_mmap-v4-0-7f038870f022@xxxxxxxxxxxxx
>
> Changes in v4:
> - Overhauled and split up patch 2 which caused errors on ppc due to
> unexported __kernel_io_end. Replaced it with a minimal s390 PCI fixup
> harness to set pdev->non_compliant_bars for ISM plus ignoring devices
> with this flag in vfio-pci. Idea for using PCI quirks came from
> Christoph Hellwig, thanks. Dropped R-bs for patch 2 accordingly.
> - Rebased on v6.10-rc5 which includes the vfio-pci mmap fault handler
> fix to the issue I stumbled over independently in v3
> - Link to v3: https://lore.kernel.org/r/20240529-vfio_pci_mmap-v3-0-cd217d019218@xxxxxxxxxxxxx
>
> Changes in v3:
> - Rebased on v6.10-rc1 requiring change to follow_pte() call
> - Use current->mm for fixup_user_fault() as seems more common
> - Collected new trailers
> - Link to v2: https://lore.kernel.org/r/20240523-vfio_pci_mmap-v2-0-0dc6c139a4f1@xxxxxxxxxxxxx
>
> Changes in v2:
> - Changed last patch to remove VFIO_PCI_MMAP instead of just enabling it
> for s390 as it is unconditionally true with s390 supporting PCI resource mmap() (Jason)
> - Collected R-bs from Jason
> - Link to v1: https://lore.kernel.org/r/20240521-vfio_pci_mmap-v1-0-2f6315e0054e@xxxxxxxxxxxxx
>
> ---
> Niklas Schnelle (3):
> s390/pci: Fix s390_mmio_read/write syscall page fault handling
> PCI: s390: Introduce pdev->non_mappable_bars and replace VFIO_PCI_MMAP
> PCI: s390: Support mmap() of PCI resources except for ISM devices
>
> arch/s390/Kconfig | 4 +---
> arch/s390/include/asm/pci.h | 3 +++
> arch/s390/pci/Makefile | 2 +-
> arch/s390/pci/pci_fixup.c | 23 +++++++++++++++++++++++
> arch/s390/pci/pci_mmio.c | 18 +++++++++++++-----
> drivers/pci/pci-sysfs.c | 4 ++++
> drivers/pci/proc.c | 4 ++++
> drivers/s390/net/ism_drv.c | 1 -
> drivers/vfio/pci/Kconfig | 4 ----
> drivers/vfio/pci/vfio_pci_core.c | 2 +-
> include/linux/pci.h | 1 +
> include/linux/pci_ids.h | 1 +
> 12 files changed, 52 insertions(+), 15 deletions(-)
Applied to pci/resource for v6.15, thanks!
I updated the subject lines to all start with "s390/pci" since that's
where all the interesting bits are and there's only a single instance
of "PCI: s390" in the history.
