This is a draft patchset for a new vendor specific VFIO driver (pds_vfio) for use with the AMD/Pensando Distributed Services Card (DSC). This driver is device type agnostic and live migration is supported as long as the underlying SR-IOV VF supports live migration on the DSC. This driver is a client of the newly introduced pds_core driver, which the latest version can be referenced at: https://lore.kernel.org/netdev/20230217225558.19837-1-shannon.nelson@xxxxxxx/ This driver will use the pds_core device and auxiliary_bus as the VFIO control path to the DSC. The pds_core device creates auxiliary_bus devices for each live migratable VF. The devices are named by their feature plus the VF PCI BDF so the auxiliary_bus driver implemented by pds_vfio can find its related VF PCI driver instance. Once this auxiliary bus connection is configured, the pds_vfio driver can send admin queue commands to the device and receive events from pds_core. An ASCII diagram of a VFIO instance looks something like this and can be used with the VFIO subsystem to provide devices VFIO and live migration support. .------. .--------------------------. | QEMU |--| VM .-------------. | '......' | | PCI driver | | | | .-------------. | | | | SR-IOV VF | | | | '-------------' | | '---------------||---------' .--------------. || |/dev/<vfio_fd>| || '--------------' || Host Userspace | || =================================================== || Host Kernel | || | || pds_core.LM.2305 <--+ .--------. || | | |vfio-pci| || | | '--------' || | | | || .------------. .-------------. || | pds_core | | pds_vfio | || '------------' '-------------' || || || || 09:00.0 09:00.1 || == PCI ==================================================||===== || || || .----------. .----------. || ,-----| PF |---------| VF |-------------------, | '----------' '----------' | VF | | DSC | data/control | | | path | ----------------------------------------------------------- The pds_vfio driver is targeted to reside in drivers/vfio/pci/pds. It makes use of and introduces new files in the common include/linux/pds include directory. Changes: v3: - Update copyright year to 2023 and use "Advanced Micro Devices, Inc." for the company name - Clarify the fact that AMD/Pensando's VFIO solution is device type agnostic, which aligns with other current VFIO solutions - Add line in drivers/vfio/pci/Makefile to build pds_vfio - Move documentation to amd sub-directory - Remove some dead code due to the pds_core implementation of listening to BIND/UNBIND events - Move a dev_dbg() to a previous patch in the series - Add implementation for vfio_migration_ops.migration_get_data_size to return the maximum possible device state size RFC to v2: https://lore.kernel.org/all/20221214232136.64220-1-brett.creeley@xxxxxxx/ - Implement state transitions for VFIO_MIGRATION_P2P flag - Improve auxiliary driver probe by returning EPROBE_DEFER when the PCI driver is not set up correctly - Add pointer to docs in Documentation/networking/device_drivers/ethernet/index.rst RFC: https://lore.kernel.org/all/20221207010705.35128-1-brett.creeley@xxxxxxx/ Brett Creeley (7): vfio/pds: Initial support for pds_vfio VFIO driver vfio/pds: Add support to register as PDS client vfio/pds: Add VFIO live migration support vfio: Commonize combine_ranges for use in other VFIO drivers vfio/pds: Add support for dirty page tracking vfio/pds: Add support for firmware recovery vfio/pds: Add Kconfig and documentation .../device_drivers/ethernet/amd/pds_vfio.rst | 88 +++ .../device_drivers/ethernet/index.rst | 1 + MAINTAINERS | 7 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/mlx5/cmd.c | 48 +- drivers/vfio/pci/pds/Kconfig | 19 + drivers/vfio/pci/pds/Makefile | 12 + drivers/vfio/pci/pds/aux_drv.c | 210 +++++++ drivers/vfio/pci/pds/aux_drv.h | 28 + drivers/vfio/pci/pds/cmds.c | 485 ++++++++++++++++ drivers/vfio/pci/pds/cmds.h | 44 ++ drivers/vfio/pci/pds/dirty.c | 541 ++++++++++++++++++ drivers/vfio/pci/pds/dirty.h | 48 ++ drivers/vfio/pci/pds/lm.c | 491 ++++++++++++++++ drivers/vfio/pci/pds/lm.h | 53 ++ drivers/vfio/pci/pds/pci_drv.c | 126 ++++ drivers/vfio/pci/pds/pci_drv.h | 14 + drivers/vfio/pci/pds/vfio_dev.c | 239 ++++++++ drivers/vfio/pci/pds/vfio_dev.h | 42 ++ drivers/vfio/vfio_main.c | 48 ++ include/linux/pds/pds_lm.h | 391 +++++++++++++ include/linux/vfio.h | 3 + 23 files changed, 2895 insertions(+), 47 deletions(-) create mode 100644 Documentation/networking/device_drivers/ethernet/amd/pds_vfio.rst create mode 100644 drivers/vfio/pci/pds/Kconfig create mode 100644 drivers/vfio/pci/pds/Makefile create mode 100644 drivers/vfio/pci/pds/aux_drv.c create mode 100644 drivers/vfio/pci/pds/aux_drv.h create mode 100644 drivers/vfio/pci/pds/cmds.c create mode 100644 drivers/vfio/pci/pds/cmds.h create mode 100644 drivers/vfio/pci/pds/dirty.c create mode 100644 drivers/vfio/pci/pds/dirty.h create mode 100644 drivers/vfio/pci/pds/lm.c create mode 100644 drivers/vfio/pci/pds/lm.h create mode 100644 drivers/vfio/pci/pds/pci_drv.c create mode 100644 drivers/vfio/pci/pds/pci_drv.h create mode 100644 drivers/vfio/pci/pds/vfio_dev.c create mode 100644 drivers/vfio/pci/pds/vfio_dev.h create mode 100644 include/linux/pds/pds_lm.h -- 2.17.1
