[cc +kvm, +lkml]
Ditto list cc comment from 18/20, and doubly so on this with updates to
the vfio uapi. Also comment below...
On Wed, 19 Dec 2018 19:52:32 +1100
Alexey Kardashevskiy <aik@xxxxxxxxx> wrote:
> POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not
> pluggable PCIe devices but still have PCIe links which are used
> for config space and MMIO. In addition to that the GPUs have 6 NVLinks
> which are connected to other GPUs and the POWER9 CPU. POWER9 chips
> have a special unit on a die called an NPU which is an NVLink2 host bus
> adapter with p2p connections to 2 to 3 GPUs, 3 or 2 NVLinks to each.
> These systems also support ATS (address translation services) which is
> a part of the NVLink2 protocol. Such GPUs also share on-board RAM
> (16GB or 32GB) to the system via the same NVLink2 so a CPU has
> cache-coherent access to a GPU RAM.
>
> This exports GPU RAM to the userspace as a new VFIO device region. This
> preregisters the new memory as device memory as it might be used for DMA.
> This inserts pfns from the fault handler as the GPU memory is not onlined
> until the vendor driver is loaded and trained the NVLinks so doing this
> earlier causes low level errors which we fence in the firmware so
> it does not hurt the host system but still better be avoided; for the same
> reason this does not map GPU RAM into the host kernel (usual thing for
> emulated access otherwise).
>
> This exports an ATSD (Address Translation Shootdown) register of NPU which
> allows TLB invalidations inside GPU for an operating system. The register
> conveniently occupies a single 64k page. It is also presented to
> the userspace as a new VFIO device region. One NPU has 8 ATSD registers,
> each of them can be used for TLB invalidation in a GPU linked to this NPU.
> This allocates one ATSD register per an NVLink bridge allowing passing
> up to 6 registers. Due to the host firmware bug (just recently fixed),
> only 1 ATSD register per NPU was actually advertised to the host system
> so this passes that alone register via the first NVLink bridge device in
> the group which is still enough as QEMU collects them all back and
> presents to the guest via vPHB to mimic the emulated NPU PHB on the host.
>
> In order to provide the userspace with the information about GPU-to-NVLink
> connections, this exports an additional capability called "tgt"
> (which is an abbreviated host system bus address). The "tgt" property
> tells the GPU its own system address and allows the guest driver to
> conglomerate the routing information so each GPU knows how to get directly
> to the other GPUs.
>
> For ATS to work, the nest MMU (an NVIDIA block in a P9 CPU) needs to
> know LPID (a logical partition ID or a KVM guest hardware ID in other
> words) and PID (a memory context ID of a userspace process, not to be
> confused with a linux pid). This assigns a GPU to LPID in the NPU and
> this is why this adds a listener for KVM on an IOMMU group. A PID comes
> via NVLink from a GPU and NPU uses a PID wildcard to pass it through.
>
> This requires coherent memory and ATSD to be available on the host as
> the GPU vendor only supports configurations with both features enabled
> and other configurations are known not to work. Because of this and
> because of the ways the features are advertised to the host system
> (which is a device tree with very platform specific properties),
> this requires enabled POWERNV platform.
>
> The V100 GPUs do not advertise any of these capabilities via the config
> space and there are more than just one device ID so this relies on
> the platform to tell whether these GPUs have special abilities such as
> NVLinks.
>
> Signed-off-by: Alexey Kardashevskiy <aik@xxxxxxxxx>
> ---
> Changes:
> v6:
> * reworked capabilities - tgt for nvlink and gpu and link-speed
> for nvlink only
>
> v5:
> * do not memremap GPU RAM for emulation, map it only when it is needed
> * allocate 1 ATSD register per NVLink bridge, if none left, then expose
> the region with a zero size
> * separate caps per device type
> * addressed AW review comments
>
> v4:
> * added nvlink-speed to the NPU bridge capability as this turned out to
> be not a constant value
> * instead of looking at the exact device ID (which also changes from system
> to system), now this (indirectly) looks at the device tree to know
> if GPU and NPU support NVLink
>
> v3:
> * reworded the commit log about tgt
> * added tracepoints (do we want them enabled for entire vfio-pci?)
> * added code comments
> * added write|mmap flags to the new regions
> * auto enabled VFIO_PCI_NVLINK2 config option
> * added 'tgt' capability to a GPU so QEMU can recreate ibm,npu and ibm,gpu
> references; there are required by the NVIDIA driver
> * keep notifier registered only for short time
> ---
> drivers/vfio/pci/Makefile | 1 +
> drivers/vfio/pci/trace.h | 102 ++++++
> drivers/vfio/pci/vfio_pci_private.h | 14 +
> include/uapi/linux/vfio.h | 38 +++
> drivers/vfio/pci/vfio_pci.c | 27 +-
> drivers/vfio/pci/vfio_pci_nvlink2.c | 482 ++++++++++++++++++++++++++++
> drivers/vfio/pci/Kconfig | 6 +
> 7 files changed, 668 insertions(+), 2 deletions(-)
> create mode 100644 drivers/vfio/pci/trace.h
> create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c
>
> diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile
> index 76d8ec0..9662c06 100644
> --- a/drivers/vfio/pci/Makefile
> +++ b/drivers/vfio/pci/Makefile
> @@ -1,5 +1,6 @@
>
> vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
> vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
> +vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o
>
> obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
> diff --git a/drivers/vfio/pci/trace.h b/drivers/vfio/pci/trace.h
> new file mode 100644
> index 0000000..b80d2d3
> --- /dev/null
> +++ b/drivers/vfio/pci/trace.h
> @@ -0,0 +1,102 @@
> +/* SPDX-License-Identifier: GPL-2.0+ */
> +/*
> + * VFIO PCI mmap/mmap_fault tracepoints
> + *
> + * Copyright (C) 2018 IBM Corp. All rights reserved.
> + * Author: Alexey Kardashevskiy <aik@xxxxxxxxx>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +#undef TRACE_SYSTEM
> +#define TRACE_SYSTEM vfio_pci
> +
> +#if !defined(_TRACE_VFIO_PCI_H) || defined(TRACE_HEADER_MULTI_READ)
> +#define _TRACE_VFIO_PCI_H
> +
> +#include <linux/tracepoint.h>
> +
> +TRACE_EVENT(vfio_pci_nvgpu_mmap_fault,
> + TP_PROTO(struct pci_dev *pdev, unsigned long hpa, unsigned long ua,
> + vm_fault_t ret),
> + TP_ARGS(pdev, hpa, ua, ret),
> +
> + TP_STRUCT__entry(
> + __field(const char *, name)
> + __field(unsigned long, hpa)
> + __field(unsigned long, ua)
> + __field(int, ret)
> + ),
> +
> + TP_fast_assign(
> + __entry->name = dev_name(&pdev->dev),
> + __entry->hpa = hpa;
> + __entry->ua = ua;
> + __entry->ret = ret;
> + ),
> +
> + TP_printk("%s: %lx -> %lx ret=%d", __entry->name, __entry->hpa,
> + __entry->ua, __entry->ret)
> +);
> +
> +TRACE_EVENT(vfio_pci_nvgpu_mmap,
> + TP_PROTO(struct pci_dev *pdev, unsigned long hpa, unsigned long ua,
> + unsigned long size, int ret),
> + TP_ARGS(pdev, hpa, ua, size, ret),
> +
> + TP_STRUCT__entry(
> + __field(const char *, name)
> + __field(unsigned long, hpa)
> + __field(unsigned long, ua)
> + __field(unsigned long, size)
> + __field(int, ret)
> + ),
> +
> + TP_fast_assign(
> + __entry->name = dev_name(&pdev->dev),
> + __entry->hpa = hpa;
> + __entry->ua = ua;
> + __entry->size = size;
> + __entry->ret = ret;
> + ),
> +
> + TP_printk("%s: %lx -> %lx size=%lx ret=%d", __entry->name, __entry->hpa,
> + __entry->ua, __entry->size, __entry->ret)
> +);
> +
> +TRACE_EVENT(vfio_pci_npu2_mmap,
> + TP_PROTO(struct pci_dev *pdev, unsigned long hpa, unsigned long ua,
> + unsigned long size, int ret),
> + TP_ARGS(pdev, hpa, ua, size, ret),
> +
> + TP_STRUCT__entry(
> + __field(const char *, name)
> + __field(unsigned long, hpa)
> + __field(unsigned long, ua)
> + __field(unsigned long, size)
> + __field(int, ret)
> + ),
> +
> + TP_fast_assign(
> + __entry->name = dev_name(&pdev->dev),
> + __entry->hpa = hpa;
> + __entry->ua = ua;
> + __entry->size = size;
> + __entry->ret = ret;
> + ),
> +
> + TP_printk("%s: %lx -> %lx size=%lx ret=%d", __entry->name, __entry->hpa,
> + __entry->ua, __entry->size, __entry->ret)
> +);
> +
> +#endif /* _TRACE_SUBSYS_H */
> +
> +#undef TRACE_INCLUDE_PATH
> +#define TRACE_INCLUDE_PATH .
> +#undef TRACE_INCLUDE_FILE
> +#define TRACE_INCLUDE_FILE trace
> +
> +/* This part must be outside protection */
> +#include <trace/define_trace.h>
> diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h
> index 93c1738..127071b 100644
> --- a/drivers/vfio/pci/vfio_pci_private.h
> +++ b/drivers/vfio/pci/vfio_pci_private.h
> @@ -163,4 +163,18 @@ static inline int vfio_pci_igd_init(struct vfio_pci_device *vdev)
> return -ENODEV;
> }
> #endif
> +#ifdef CONFIG_VFIO_PCI_NVLINK2
> +extern int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev);
> +extern int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev);
> +#else
> +static inline int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev)
> +{
> + return -ENODEV;
> +}
> +
> +static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
> +{
> + return -ENODEV;
> +}
> +#endif
> #endif /* VFIO_PCI_PRIVATE_H */
> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index 8131028..22b825c 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -353,6 +353,21 @@ struct vfio_region_gfx_edid {
> #define VFIO_DEVICE_GFX_LINK_STATE_DOWN 2
> };
>
> +/*
> + * 10de vendor sub-type
> + *
> + * NVIDIA GPU NVlink2 RAM is coherent RAM mapped onto the host address space.
> + */
> +#define VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM (1)
> +
> +/*
> + * 1014 vendor sub-type
> + *
> + * IBM NPU NVlink2 ATSD (Address Translation Shootdown) register of NPU
> + * to do TLB invalidation on a GPU.
> + */
> +#define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1)
> +
> /*
> * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped
> * which allows direct access to non-MSIX registers which happened to be within
> @@ -363,6 +378,29 @@ struct vfio_region_gfx_edid {
> */
> #define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3
>
> +/*
> + * Capability with compressed real address (aka SSA - small system address)
> + * where GPU RAM is mapped on a system bus. Used by a GPU for DMA routing.
> + */
> +#define VFIO_REGION_INFO_CAP_NVLINK2_SSATGT 4
> +
> +struct vfio_region_info_cap_nvlink2_ssatgt {
> + struct vfio_info_cap_header header;
> + __u64 tgt;
> +};
> +
> +/*
> + * Capability with compressed real address (aka SSA - small system address),
> + * used to match the NVLink bridge with a GPU. Also contains a link speed.
> + */
Comments carried over from previous definitions are no longer
accurate. Thanks,
Alex
> +#define VFIO_REGION_INFO_CAP_NVLINK2_LNKSPD 5
> +
> +struct vfio_region_info_cap_nvlink2_lnkspd {
> + struct vfio_info_cap_header header;
> + __u32 link_speed;
> + __u32 __pad;
> +};
> +
> /**
> * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9,
> * struct vfio_irq_info)
> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
> index 6cb70cf..67c03f2 100644
> --- a/drivers/vfio/pci/vfio_pci.c
> +++ b/drivers/vfio/pci/vfio_pci.c
> @@ -302,14 +302,37 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
> if (ret) {
> dev_warn(&vdev->pdev->dev,
> "Failed to setup Intel IGD regions\n");
> - vfio_pci_disable(vdev);
> - return ret;
> + goto disable_exit;
> + }
> + }
> +
> + if (pdev->vendor == PCI_VENDOR_ID_NVIDIA &&
> + IS_ENABLED(CONFIG_VFIO_PCI_NVLINK2)) {
> + ret = vfio_pci_nvdia_v100_nvlink2_init(vdev);
> + if (ret && ret != -ENODEV) {
> + dev_warn(&vdev->pdev->dev,
> + "Failed to setup NVIDIA NV2 RAM region\n");
> + goto disable_exit;
> + }
> + }
> +
> + if (pdev->vendor == PCI_VENDOR_ID_IBM &&
> + IS_ENABLED(CONFIG_VFIO_PCI_NVLINK2)) {
> + ret = vfio_pci_ibm_npu2_init(vdev);
> + if (ret && ret != -ENODEV) {
> + dev_warn(&vdev->pdev->dev,
> + "Failed to setup NVIDIA NV2 ATSD region\n");
> + goto disable_exit;
> }
> }
>
> vfio_pci_probe_mmaps(vdev);
>
> return 0;
> +
> +disable_exit:
> + vfio_pci_disable(vdev);
> + return ret;
> }
>
> static void vfio_pci_disable(struct vfio_pci_device *vdev)
> diff --git a/drivers/vfio/pci/vfio_pci_nvlink2.c b/drivers/vfio/pci/vfio_pci_nvlink2.c
> new file mode 100644
> index 0000000..054a2cf
> --- /dev/null
> +++ b/drivers/vfio/pci/vfio_pci_nvlink2.c
> @@ -0,0 +1,482 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * VFIO PCI NVIDIA Whitherspoon GPU support a.k.a. NVLink2.
> + *
> + * Copyright (C) 2018 IBM Corp. All rights reserved.
> + * Author: Alexey Kardashevskiy <aik@xxxxxxxxx>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * Register an on-GPU RAM region for cacheable access.
> + *
> + * Derived from original vfio_pci_igd.c:
> + * Copyright (C) 2016 Red Hat, Inc. All rights reserved.
> + * Author: Alex Williamson <alex.williamson@xxxxxxxxxx>
> + */
> +
> +#include <linux/io.h>
> +#include <linux/pci.h>
> +#include <linux/uaccess.h>
> +#include <linux/vfio.h>
> +#include <linux/sched/mm.h>
> +#include <linux/mmu_context.h>
> +#include <asm/kvm_ppc.h>
> +#include "vfio_pci_private.h"
> +
> +#define CREATE_TRACE_POINTS
> +#include "trace.h"
> +
> +EXPORT_TRACEPOINT_SYMBOL_GPL(vfio_pci_nvgpu_mmap_fault);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(vfio_pci_nvgpu_mmap);
> +EXPORT_TRACEPOINT_SYMBOL_GPL(vfio_pci_npu2_mmap);
> +
> +struct vfio_pci_nvgpu_data {
> + unsigned long gpu_hpa; /* GPU RAM physical address */
> + unsigned long gpu_tgt; /* TGT address of corresponding GPU RAM */
> + unsigned long useraddr; /* GPU RAM userspace address */
> + unsigned long size; /* Size of the GPU RAM window (usually 128GB) */
> + struct mm_struct *mm;
> + struct mm_iommu_table_group_mem_t *mem; /* Pre-registered RAM descr. */
> + struct pci_dev *gpdev;
> + struct notifier_block group_notifier;
> +};
> +
> +static size_t vfio_pci_nvgpu_rw(struct vfio_pci_device *vdev,
> + char __user *buf, size_t count, loff_t *ppos, bool iswrite)
> +{
> + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS;
> + struct vfio_pci_nvgpu_data *data = vdev->region[i].data;
> + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
> + loff_t posaligned = pos & PAGE_MASK, posoff = pos & ~PAGE_MASK;
> + size_t sizealigned;
> + void __iomem *ptr;
> +
> + if (pos >= vdev->region[i].size)
> + return -EINVAL;
> +
> + count = min(count, (size_t)(vdev->region[i].size - pos));
> +
> + /*
> + * We map only a bit of GPU RAM for a short time instead of mapping it
> + * for the guest lifetime as:
> + *
> + * 1) we do not know GPU RAM size, only aperture which is 4-8 times
> + * bigger than actual RAM size (16/32GB RAM vs. 128GB aperture);
> + * 2) mapping GPU RAM allows CPU to prefetch and if this happens
> + * before NVLink bridge is reset (which fences GPU RAM),
> + * hardware management interrupts (HMI) might happen, this
> + * will freeze NVLink bridge.
> + *
> + * This is not fast path anyway.
> + */
> + sizealigned = _ALIGN_UP(posoff + count, PAGE_SIZE);
> + ptr = ioremap_cache(data->gpu_hpa + posaligned, sizealigned);
> + if (!ptr)
> + return -EFAULT;
> +
> + if (iswrite) {
> + if (copy_from_user(ptr + posoff, buf, count))
> + count = -EFAULT;
> + else
> + *ppos += count;
> + } else {
> + if (copy_to_user(buf, ptr + posoff, count))
> + count = -EFAULT;
> + else
> + *ppos += count;
> + }
> +
> + iounmap(ptr);
> +
> + return count;
> +}
> +
> +static void vfio_pci_nvgpu_release(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region)
> +{
> + struct vfio_pci_nvgpu_data *data = region->data;
> + long ret;
> +
> + /* If there were any mappings at all... */
> + if (data->mm) {
> + ret = mm_iommu_put(data->mm, data->mem);
> + WARN_ON(ret);
> +
> + mmdrop(data->mm);
> + }
> +
> + vfio_unregister_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
> + &data->group_notifier);
> +
> + pnv_npu2_unmap_lpar_dev(data->gpdev);
> +
> + kfree(data);
> +}
> +
> +static vm_fault_t vfio_pci_nvgpu_mmap_fault(struct vm_fault *vmf)
> +{
> + vm_fault_t ret;
> + struct vm_area_struct *vma = vmf->vma;
> + struct vfio_pci_region *region = vma->vm_private_data;
> + struct vfio_pci_nvgpu_data *data = region->data;
> + unsigned long vmf_off = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
> + unsigned long nv2pg = data->gpu_hpa >> PAGE_SHIFT;
> + unsigned long vm_pgoff = vma->vm_pgoff &
> + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
> + unsigned long pfn = nv2pg + vm_pgoff + vmf_off;
> +
> + ret = vmf_insert_pfn(vma, vmf->address, pfn);
> + trace_vfio_pci_nvgpu_mmap_fault(data->gpdev, pfn << PAGE_SHIFT,
> + vmf->address, ret);
> +
> + return ret;
> +}
> +
> +static const struct vm_operations_struct vfio_pci_nvgpu_mmap_vmops = {
> + .fault = vfio_pci_nvgpu_mmap_fault,
> +};
> +
> +static int vfio_pci_nvgpu_mmap(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region, struct vm_area_struct *vma)
> +{
> + int ret;
> + struct vfio_pci_nvgpu_data *data = region->data;
> +
> + if (data->useraddr)
> + return -EPERM;
> +
> + if (vma->vm_end - vma->vm_start > data->size)
> + return -EINVAL;
> +
> + vma->vm_private_data = region;
> + vma->vm_flags |= VM_PFNMAP;
> + vma->vm_ops = &vfio_pci_nvgpu_mmap_vmops;
> +
> + /*
> + * Calling mm_iommu_newdev() here once as the region is not
> + * registered yet and therefore right initialization will happen now.
> + * Other places will use mm_iommu_find() which returns
> + * registered @mem and does not go gup().
> + */
> + data->useraddr = vma->vm_start;
> + data->mm = current->mm;
> +
> + atomic_inc(&data->mm->mm_count);
> + ret = (int) mm_iommu_newdev(data->mm, data->useraddr,
> + (vma->vm_end - vma->vm_start) >> PAGE_SHIFT,
> + data->gpu_hpa, &data->mem);
> +
> + trace_vfio_pci_nvgpu_mmap(vdev->pdev, data->gpu_hpa, data->useraddr,
> + vma->vm_end - vma->vm_start, ret);
> +
> + return ret;
> +}
> +
> +static int vfio_pci_nvgpu_add_capability(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region, struct vfio_info_cap *caps)
> +{
> + struct vfio_pci_nvgpu_data *data = region->data;
> + struct vfio_region_info_cap_nvlink2_ssatgt cap = { 0 };
> +
> + cap.header.id = VFIO_REGION_INFO_CAP_NVLINK2_SSATGT;
> + cap.header.version = 1;
> + cap.tgt = data->gpu_tgt;
> +
> + return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
> +}
> +
> +static const struct vfio_pci_regops vfio_pci_nvgpu_regops = {
> + .rw = vfio_pci_nvgpu_rw,
> + .release = vfio_pci_nvgpu_release,
> + .mmap = vfio_pci_nvgpu_mmap,
> + .add_capability = vfio_pci_nvgpu_add_capability,
> +};
> +
> +static int vfio_pci_nvgpu_group_notifier(struct notifier_block *nb,
> + unsigned long action, void *opaque)
> +{
> + struct kvm *kvm = opaque;
> + struct vfio_pci_nvgpu_data *data = container_of(nb,
> + struct vfio_pci_nvgpu_data,
> + group_notifier);
> +
> + if (action == VFIO_GROUP_NOTIFY_SET_KVM && kvm &&
> + pnv_npu2_map_lpar_dev(data->gpdev,
> + kvm->arch.lpid, MSR_DR | MSR_PR))
> + return NOTIFY_BAD;
> +
> + return NOTIFY_OK;
> +}
> +
> +int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev)
> +{
> + int ret;
> + u64 reg[2];
> + u64 tgt = 0;
> + struct device_node *npu_node, *mem_node;
> + struct pci_dev *npu_dev;
> + struct vfio_pci_nvgpu_data *data;
> + uint32_t mem_phandle = 0;
> + unsigned long events = VFIO_GROUP_NOTIFY_SET_KVM;
> +
> + /*
> + * PCI config space does not tell us about NVLink presense but
> + * platform does, use this.
> + */
> + npu_dev = pnv_pci_get_npu_dev(vdev->pdev, 0);
> + if (!npu_dev)
> + return -ENODEV;
> +
> + npu_node = pci_device_to_OF_node(npu_dev);
> + if (!npu_node)
> + return -EINVAL;
> +
> + if (of_property_read_u32(npu_node, "memory-region", &mem_phandle))
> + return -EINVAL;
> +
> + mem_node = of_find_node_by_phandle(mem_phandle);
> + if (!mem_node)
> + return -EINVAL;
> +
> + if (of_property_read_variable_u64_array(mem_node, "reg", reg,
> + ARRAY_SIZE(reg), ARRAY_SIZE(reg)) !=
> + ARRAY_SIZE(reg))
> + return -EINVAL;
> +
> + if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) {
> + dev_warn(&vdev->pdev->dev, "No ibm,device-tgt-addr found\n");
> + return -EFAULT;
> + }
> +
> + data = kzalloc(sizeof(*data), GFP_KERNEL);
> + if (!data)
> + return -ENOMEM;
> +
> + data->gpu_hpa = reg[0];
> + data->gpu_tgt = tgt;
> + data->size = reg[1];
> +
> + dev_dbg(&vdev->pdev->dev, "%lx..%lx\n", data->gpu_hpa,
> + data->gpu_hpa + data->size - 1);
> +
> + data->gpdev = vdev->pdev;
> + data->group_notifier.notifier_call = vfio_pci_nvgpu_group_notifier;
> +
> + ret = vfio_register_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
> + &events, &data->group_notifier);
> + if (ret)
> + goto free_exit;
> +
> + /*
> + * We have just set KVM, we do not need the listener anymore.
> + * Also, keeping it registered means that if more than one GPU is
> + * assigned, we will get several similar notifiers notifying about
> + * the same device again which does not help with anything.
> + */
> + vfio_unregister_notifier(&data->gpdev->dev, VFIO_GROUP_NOTIFY,
> + &data->group_notifier);
> +
> + ret = vfio_pci_register_dev_region(vdev,
> + PCI_VENDOR_ID_NVIDIA | VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
> + VFIO_REGION_SUBTYPE_NVIDIA_NVLINK2_RAM,
> + &vfio_pci_nvgpu_regops,
> + data->size,
> + VFIO_REGION_INFO_FLAG_READ |
> + VFIO_REGION_INFO_FLAG_WRITE |
> + VFIO_REGION_INFO_FLAG_MMAP,
> + data);
> + if (ret)
> + goto free_exit;
> +
> + return 0;
> +free_exit:
> + kfree(data);
> +
> + return ret;
> +}
> +
> +/*
> + * IBM NPU2 bridge
> + */
> +struct vfio_pci_npu2_data {
> + void *base; /* ATSD register virtual address, for emulated access */
> + unsigned long mmio_atsd; /* ATSD physical address */
> + unsigned long gpu_tgt; /* TGT address of corresponding GPU RAM */
> + unsigned int link_speed; /* The link speed from DT's ibm,nvlink-speed */
> +};
> +
> +static size_t vfio_pci_npu2_rw(struct vfio_pci_device *vdev,
> + char __user *buf, size_t count, loff_t *ppos, bool iswrite)
> +{
> + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS;
> + struct vfio_pci_npu2_data *data = vdev->region[i].data;
> + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK;
> +
> + if (pos >= vdev->region[i].size)
> + return -EINVAL;
> +
> + count = min(count, (size_t)(vdev->region[i].size - pos));
> +
> + if (iswrite) {
> + if (copy_from_user(data->base + pos, buf, count))
> + return -EFAULT;
> + } else {
> + if (copy_to_user(buf, data->base + pos, count))
> + return -EFAULT;
> + }
> + *ppos += count;
> +
> + return count;
> +}
> +
> +static int vfio_pci_npu2_mmap(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region, struct vm_area_struct *vma)
> +{
> + int ret;
> + struct vfio_pci_npu2_data *data = region->data;
> + unsigned long req_len = vma->vm_end - vma->vm_start;
> +
> + if (req_len != PAGE_SIZE)
> + return -EINVAL;
> +
> + vma->vm_flags |= VM_PFNMAP;
> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> +
> + ret = remap_pfn_range(vma, vma->vm_start, data->mmio_atsd >> PAGE_SHIFT,
> + req_len, vma->vm_page_prot);
> + trace_vfio_pci_npu2_mmap(vdev->pdev, data->mmio_atsd, vma->vm_start,
> + vma->vm_end - vma->vm_start, ret);
> +
> + return ret;
> +}
> +
> +static void vfio_pci_npu2_release(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region)
> +{
> + struct vfio_pci_npu2_data *data = region->data;
> +
> + memunmap(data->base);
> + kfree(data);
> +}
> +
> +static int vfio_pci_npu2_add_capability(struct vfio_pci_device *vdev,
> + struct vfio_pci_region *region, struct vfio_info_cap *caps)
> +{
> + struct vfio_pci_npu2_data *data = region->data;
> + struct vfio_region_info_cap_nvlink2_ssatgt captgt = { 0 };
> + struct vfio_region_info_cap_nvlink2_lnkspd capspd = { 0 };
> + int ret;
> +
> + captgt.header.id = VFIO_REGION_INFO_CAP_NVLINK2_SSATGT;
> + captgt.header.version = 1;
> + captgt.tgt = data->gpu_tgt;
> +
> + capspd.header.id = VFIO_REGION_INFO_CAP_NVLINK2_LNKSPD;
> + capspd.header.version = 1;
> + capspd.link_speed = data->link_speed;
> +
> + ret = vfio_info_add_capability(caps, &captgt.header, sizeof(captgt));
> + if (ret)
> + return ret;
> +
> + return vfio_info_add_capability(caps, &capspd.header, sizeof(capspd));
> +}
> +
> +static const struct vfio_pci_regops vfio_pci_npu2_regops = {
> + .rw = vfio_pci_npu2_rw,
> + .mmap = vfio_pci_npu2_mmap,
> + .release = vfio_pci_npu2_release,
> + .add_capability = vfio_pci_npu2_add_capability,
> +};
> +
> +int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
> +{
> + int ret;
> + struct vfio_pci_npu2_data *data;
> + struct device_node *nvlink_dn;
> + u32 nvlink_index = 0;
> + struct pci_dev *npdev = vdev->pdev;
> + struct device_node *npu_node = pci_device_to_OF_node(npdev);
> + struct pci_controller *hose = pci_bus_to_host(npdev->bus);
> + u64 mmio_atsd = 0;
> + u64 tgt = 0;
> + u32 link_speed = 0xff;
> +
> + /*
> + * PCI config space does not tell us about NVLink presense but
> + * platform does, use this.
> + */
> + if (!pnv_pci_get_gpu_dev(vdev->pdev))
> + return -ENODEV;
> +
> + /*
> + * NPU2 normally has 8 ATSD registers (for concurrency) and 6 links
> + * so we can allocate one register per link, using nvlink index as
> + * a key.
> + * There is always at least one ATSD register so as long as at least
> + * NVLink bridge #0 is passed to the guest, ATSD will be available.
> + */
> + nvlink_dn = of_parse_phandle(npdev->dev.of_node, "ibm,nvlink", 0);
> + if (WARN_ON(of_property_read_u32(nvlink_dn, "ibm,npu-link-index",
> + &nvlink_index)))
> + return -ENODEV;
> +
> + if (of_property_read_u64_index(hose->dn, "ibm,mmio-atsd", nvlink_index,
> + &mmio_atsd)) {
> + dev_warn(&vdev->pdev->dev, "No available ATSD found\n");
> + mmio_atsd = 0;
> + }
> +
> + if (of_property_read_u64(npu_node, "ibm,device-tgt-addr", &tgt)) {
> + dev_warn(&vdev->pdev->dev, "No ibm,device-tgt-addr found\n");
> + return -EFAULT;
> + }
> +
> + if (of_property_read_u32(npu_node, "ibm,nvlink-speed", &link_speed)) {
> + dev_warn(&vdev->pdev->dev, "No ibm,nvlink-speed found\n");
> + return -EFAULT;
> + }
> +
> + data = kzalloc(sizeof(*data), GFP_KERNEL);
> + if (!data)
> + return -ENOMEM;
> +
> + data->mmio_atsd = mmio_atsd;
> + data->gpu_tgt = tgt;
> + data->link_speed = link_speed;
> + if (data->mmio_atsd) {
> + data->base = memremap(data->mmio_atsd, SZ_64K, MEMREMAP_WT);
> + if (!data->base) {
> + ret = -ENOMEM;
> + goto free_exit;
> + }
> + }
> +
> + /*
> + * We want to expose the capability even if this specific NVLink
> + * did not get its own ATSD register because capabilities
> + * belong to VFIO regions and normally there will be ATSD register
> + * assigned to the NVLink bridge.
> + */
> + ret = vfio_pci_register_dev_region(vdev,
> + PCI_VENDOR_ID_IBM |
> + VFIO_REGION_TYPE_PCI_VENDOR_TYPE,
> + VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD,
> + &vfio_pci_npu2_regops,
> + data->mmio_atsd ? PAGE_SIZE : 0,
> + VFIO_REGION_INFO_FLAG_READ |
> + VFIO_REGION_INFO_FLAG_WRITE |
> + VFIO_REGION_INFO_FLAG_MMAP,
> + data);
> + if (ret)
> + goto free_exit;
> +
> + return 0;
> +
> +free_exit:
> + kfree(data);
> +
> + return ret;
> +}
> diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
> index 42dc1d3..d0f8e4f 100644
> --- a/drivers/vfio/pci/Kconfig
> +++ b/drivers/vfio/pci/Kconfig
> @@ -38,3 +38,9 @@ config VFIO_PCI_IGD
> and LPC bridge config space.
>
> To enable Intel IGD assignment through vfio-pci, say Y.
> +
> +config VFIO_PCI_NVLINK2
> + def_bool y
> + depends on VFIO_PCI && PPC_POWERNV
> + help
> + VFIO PCI support for P9 Witherspoon machine with NVIDIA V100 GPUs
