On 12/7/2022 3:06 AM, Brett Creeley wrote:
This is a first draft patchset for a new vendor specific VFIO driver for use with the AMD/Pensando Distributed Services Card (DSC). This driver (pds_vfio) is a client of the newly introduced pds_core driver. Reference to the pds_core patchset: https://lore.kernel.org/netdev/20221207004443.33779-1-shannon.nelson@xxxxxxx/ AMD/Pensando already supports a NVMe VF device (1dd8:1006) in the Distributed Services Card (DSC). This patchset adds the new pds_vfio driver in order to support NVMe VF live migration. This driver will use the pds_core device and auxiliary_bus as the VFIO control path to the DSC. The pds_core device creates auxiliary_bus devices for each live migratable VF. The devices are named by their feature plus the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find its related VF PCI driver instance. Once this auxiliary bus connection is configured, the pds_vfio driver can send admin queue commands to the device and receive events from pds_core. An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide devices VFIO and live migration support. .------. .--------------------------. | QEMU |--| VM .-------------. | '......' | | nvme driver | | | | .-------------. | | | | SR-IOV VF | | | | '-------------' | | '---------------||---------' .--------------. || |/dev/<vfio_fd>| || '--------------' || Host Userspace | || =================================================== || Host Kernel | || | || pds_core.LM.2305 <--+ .--------. || | | |vfio-pci| || | | '--------' || | | | || .------------. .-------------. || | pds_core | | pds_vfio | || '------------' '-------------' || || || || 09:00.0 09:00.1 || == PCI ==================================================||===== || || || .----------. .----------. || ,-----| PF |---------| VF |-------------------, | '----------' '----------' | nvme | | DSC | data/control | | | path | -----------------------------------------------------------
Hi Brett, what is the class code of the pds_core device ? I see that pds_vfio class_code is PCI_CLASS_STORAGE_EXPRESS.
The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory. Brett Creeley (7): pds_vfio: Initial support for pds_vfio VFIO driver pds_vfio: Add support to register as PDS client pds_vfio: Add VFIO live migration support vfio: Commonize combine_ranges for use in other VFIO drivers pds_vfio: Add support for dirty page tracking pds_vfio: Add support for firmware recovery pds_vfio: Add documentation files .../ethernet/pensando/pds_vfio.rst | 88 +++ drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/mlx5/cmd.c | 48 +- drivers/vfio/pci/pds/Kconfig | 10 + drivers/vfio/pci/pds/Makefile | 12 + drivers/vfio/pci/pds/aux_drv.c | 216 +++++++ drivers/vfio/pci/pds/aux_drv.h | 30 + drivers/vfio/pci/pds/cmds.c | 486 ++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 44 ++ drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 49 ++ drivers/vfio/pci/pds/lm.c | 484 ++++++++++++++++ drivers/vfio/pci/pds/lm.h | 53 ++ drivers/vfio/pci/pds/pci_drv.c | 134 +++++ drivers/vfio/pci/pds/pci_drv.h | 9 + drivers/vfio/pci/pds/vfio_dev.c | 238 ++++++++ drivers/vfio/pci/pds/vfio_dev.h | 42 ++ drivers/vfio/vfio_main.c | 48 ++ include/linux/pds/pds_core_if.h | 1 + include/linux/pds/pds_lm.h | 356 ++++++++++++ include/linux/vfio.h | 3 + 21 files changed, 2847 insertions(+), 47 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/pensando/pds_vfio.rst create mode 100644 drivers/vfio/pci/pds/Kconfig create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/aux_drv.c create mode 100644 drivers/vfio/pci/pds/aux_drv.h create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h create mode 100644 include/linux/pds/pds_lm.h -- 2.17.1
