Hi Alex and Cornelia, This series split the vfio_pci driver into 2 parts: pci drivers and a subsystem driver that will also be library of code. The main pci driver, vfio_pci.ko will be used as before and it will bind to the subsystem driver vfio_pci_core.ko to register to the VFIO subsystem. New vendor vfio pci drivers were introduced in this series are: - igd_vfio_pci.ko - nvlink2gpu_vfio_pci.ko - npu2_vfio_pci.ko These drivers also will bind to the subsystem driver vfio_pci_core.ko to register to the VFIO subsystem. These drivers will also add vendor specific extensions that are relevant only for the devices that will be driven by them. This is a typical Linux subsystem framework behaviour. This series is coming to solve some of the issues that were raised in the previous attempt for extending vfio-pci for vendor specific functionality: https://lkml.org/lkml/2020/5/17/376 by Yan Zhao. This solution is also deterministic in a sense that when a user will bind to a vendor specific vfio-pci driver, it will get all the special goodies of the HW. Non-common code will be pushed only to the specific vfio_pci driver. This subsystem framework will also ease on adding new vendor specific functionality to VFIO devices in the future by allowing another module to provide the pci_driver that can setup number of details before registering to VFIO subsystem (such as inject its own operations). Below we can see the proposed changes: +-------------------------------------------------------------------+ | | | VFIO | | | +-------------------------------------------------------------------+ +-------------------------------------------------------------------+ | | | VFIO_PCI_CORE | | | +-------------------------------------------------------------------+ +--------------+ +------------+ +-------------+ +-------------------+ | | | | | | | | | | | | | | | | | VFIO_PCI | |IGD_VFIO_PCI| |NPU2_VFIO_PCI| |NVLINK2GPU_VFIO_PCI| | | | | | | | | | | | | | | | | +--------------+ +------------+ +-------------+ +-------------------+ Patches (1/9) - (4/9) introduce the above changes for vfio_pci and vfio_pci_core. Patches (6/9) and (7/9) are a preparation for adding nvlink2 related vfio pci drivers. Patch (8/9) introduce new npu2_vfio_pci and nvlink2gpu_vfio_pci drivers that will drive NVLINK2 capable devices exist today (IBMs emulated PCI managment device and NVIDIAs NVLINK2 capable GPUs). These drivers add vendor specific functionality that is related to the IBM NPU2 unit which is an NVLink2 host bus adapter and to capable NVIDIA GPUs. Patch (9/9) introduce new igd_vfio_pci driver for adding special extensions for INTELs Graphics card (GVT-d). All new 3 vendor specific vfio_pci drivers can be extended easily and new vendor specific functionality might be added to the needed vendor drivers. In case there is a vendor specific vfio_pci driver for the device that a user would like to use, this driver should be bounded to that device. Otherwise, vfio_pci.ko should be bounded to it (backward compatability is also supported). This framework will allow adding more HW specific features such as Live Migration in the future by extending existing vendor vfio_pci drivers or creating new ones (e.g. mlx5_vfio_pci.ko that will drive live migration for mlx5 PCI devices and mlx5_snap_vfio_pci.ko that will drive live migration for mlx5_snap devices such as mlx5 NVMe and Virtio-BLK SNAP devices). Testing: 1. vfio_pci.ko was tested by using 2 VirtIO-BLK physical functions based on NVIDIAs Bluefield-2 SNAP technology. These 2 PCI functions were passed to a QEMU based VM after binding to vfio_pci.ko and basic FIO read/write was issued in the VM to each exposed block device (vda, vdb). 2. igd_vfio_pci.ko was only compiled and loaded/unloaded successfully on x86 server. 3. npu2_vfio_pci.ko and nvlink2gpu_vfio_pci.KO were successfully compiled and loaded/unloaded on P9 server + vfio probe/remove of devices (without QEMU/VM). Note: After this series will be approved, a new discovery/matching mechanism should be introduced in order to help users to decide which driver should be bounded to which device. changes from v2: - Use container_of approach between core and vfio_pci drivers. - Comment on private/public attributes for the core structure to ease on code maintanance. - rename core structure to vfio_pci_core_device. - create 3 new vendor drivers (igd_vfio_pci, npu2_vfio_pci, nvlink2gpu_vfio_pci) and preserve backward compatibility. - rebase on top of Linux 5.11 tag. - remove patches that were accepted as stand-alone. - removed mlx5_vfio_pci driver from this series (3 vendor drivers will be enough for emphasizing the approch). changes from v1: - Create a private and public vfio-pci structures (From Alex) - register to vfio core directly from vfio-pci-core (for now, expose minimal public funcionality to vfio pci drivers. This will remove the implicit behaviour from v1. More power to the drivers can be added in the future) - Add patch 3/9 to emphasize the needed extension for LM feature (From Cornelia) - take/release refcount for the pci module during core open/release - update nvlink, igd and zdev to PowerNV, X86 and s390 extensions for vfio-pci core - fix segfault bugs in current vfio-pci zdev code Max Gurtovoy (9): vfio-pci: rename vfio_pci.c to vfio_pci_core.c vfio-pci: rename vfio_pci_private.h to vfio_pci_core.h vfio-pci: rename vfio_pci_device to vfio_pci_core_device vfio-pci: introduce vfio_pci_core subsystem driver vfio/pci: introduce vfio_pci_device structure vfio-pci-core: export vfio_pci_register_dev_region function vfio/pci_core: split nvlink2 to nvlink2gpu and npu2 vfio/pci: export nvlink2 support into vendor vfio_pci drivers vfio/pci: export igd support into vendor vfio_pci driver drivers/vfio/pci/Kconfig | 53 +- drivers/vfio/pci/Makefile | 20 +- .../pci/{vfio_pci_igd.c => igd_vfio_pci.c} | 159 +- drivers/vfio/pci/igd_vfio_pci.h | 24 + drivers/vfio/pci/npu2_trace.h | 50 + drivers/vfio/pci/npu2_vfio_pci.c | 364 +++ drivers/vfio/pci/npu2_vfio_pci.h | 24 + .../vfio/pci/{trace.h => nvlink2gpu_trace.h} | 27 +- ...io_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} | 296 +- drivers/vfio/pci/nvlink2gpu_vfio_pci.h | 24 + drivers/vfio/pci/vfio_pci.c | 2433 ++--------------- drivers/vfio/pci/vfio_pci_config.c | 70 +- drivers/vfio/pci/vfio_pci_core.c | 2245 +++++++++++++++ drivers/vfio/pci/vfio_pci_core.h | 242 ++ drivers/vfio/pci/vfio_pci_intrs.c | 42 +- drivers/vfio/pci/vfio_pci_private.h | 228 -- drivers/vfio/pci/vfio_pci_rdwr.c | 18 +- drivers/vfio/pci/vfio_pci_zdev.c | 12 +- 18 files changed, 3528 insertions(+), 2803 deletions(-) rename drivers/vfio/pci/{vfio_pci_igd.c => igd_vfio_pci.c} (58%) create mode 100644 drivers/vfio/pci/igd_vfio_pci.h create mode 100644 drivers/vfio/pci/npu2_trace.h create mode 100644 drivers/vfio/pci/npu2_vfio_pci.c create mode 100644 drivers/vfio/pci/npu2_vfio_pci.h rename drivers/vfio/pci/{trace.h => nvlink2gpu_trace.h} (72%) rename drivers/vfio/pci/{vfio_pci_nvlink2.c => nvlink2gpu_vfio_pci.c} (57%) create mode 100644 drivers/vfio/pci/nvlink2gpu_vfio_pci.h create mode 100644 drivers/vfio/pci/vfio_pci_core.c create mode 100644 drivers/vfio/pci/vfio_pci_core.h delete mode 100644 drivers/vfio/pci/vfio_pci_private.h -- 2.25.4
