The actual code is independent of the stage 2 driver code submission that adds support for SVM, ENQCMD(S), PASID, and shared workqueues. This code series will support dedicated workqueue on a guest with no vIOMMU. A new device type "mdev" is introduced for the idxd driver. This allows the wq to be dedicated to the usage of a VFIO mediated device (mdev). Once the work queue (wq) is enabled, an uuid generated by the user can be added to the wq through the uuid sysfs attribute for the wq. After the association, a mdev can be created using this UUID. The mdev driver code will associate the uuid and setup the mdev on the driver side. When the create operation is successful, the uuid can be passed to qemu. When the guest boots up, it should discover a DSA device when doing PCI discovery. For example: 1. Enable wq with “mdev” wq type 2. A user generated UUID is associated with a wq: echo $UUID > /sys/bus/dsa/devices/wq0.0/uuid 3. The uuid is written to the mdev class sysfs path: echo $UUID > /sys/class/mdev_bus/0000\:00\:0a.0/mdev_supported_types/idxd-wq/create 4. Pass the following parameter to qemu: "-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:0a.0/$UUID" Since the mdev is an emulated device with a single wq, the guest will see a DSA device with a single wq. With no vIOMMU support, the behavior will be the same as the stage 1 driver running with no IOMMU turned on on the bare metal host. The difference is that the wq exported through mdev will have the read only config bit set for configuration. This means that the device does not require the typical configuration. After enabling the device, the user must set the WQ type and name. That is all is necessary to enable the WQ and start using it. The single wq configuration is not the only way to create the mdev. Multi wq support for mdev will be in the future works. The mdev utilizes Interrupt Message Store or IMS[3] instead of MSIX for interrupts for the guest. This preserves MSIX for host usages and also allows a significantly larger number of interrupt vectors for guest usage. The idxd driver implements IMS as on-device memory mapped unified storage. Each interrupt message is stored as a DWORD size data payload and a 64-bit address (same as MSI-X). Access to the IMS is through the host idxd driver. All the IMS interrupt messages are stored in the remappable format. Hence, if the driver enables IMS, interrupt remapping is also enabled by default. This patchset extends the existing platfrom-msi.c which already provides a generic mechanism to support non-PCI compliant MSI interrupts for platform devices to provide the IMS infrastructure. More details about IMS, its implementation in the the kernel, common misconceptions about IMS and the basic driver changes required to support IMS can be found under Documentations/interrupt_message_store.txt [1]: https://lore.kernel.org/lkml/157965011794.73301.15960052071729101309.stgit@xxxxxxxxxxxxxxxxxxxxxxxxxx/ [2]: https://software.intel.com/en-us/articles/intel-sdm [3]: https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification [4]: https://software.intel.com/en-us/download/intel-data-streaming-accelerator-preliminary-architecture-specification [5]: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator [6]: https://intel.github.io/idxd/ [7]: https://github.com/intel/idxd-driver idxd-stage3 --- Dave Jiang (5): dmaengine: idxd: add config support for readonly devices dmaengine: idxd: add IMS support in base driver dmaengine: idxd: add device support functions in prep for mdev dmaengine: idxd: add support for VFIO mediated device dmaengine: idxd: add error notification from host driver to mediated device Jing Lin (1): dmaengine: idxd: add ABI documentation for mediated device support Lu Baolu (2): vfio/mdev: Add a member for iommu domain in mdev_device vfio/type1: Save domain when attach domain to mdev Megha Dey (7): drivers/base: Introduce platform_msi_ops drivers/base: Introduce a new platform-msi list drivers/base: Allocate/free platform-msi interrupts by group drivers/base: Add support for a new IMS irq domain ims-msi: Add mask/unmask routines ims-msi: Enable IMS interrupts Documentation: Interrupt Message store Documentation/ABI/stable/sysfs-driver-dma-idxd | 18 Documentation/ims-howto.rst | 210 +++ arch/x86/include/asm/hw_irq.h | 7 arch/x86/include/asm/irq_remapping.h | 6 drivers/base/Kconfig | 9 drivers/base/Makefile | 1 drivers/base/core.c | 1 drivers/base/ims-msi.c | 162 ++ drivers/base/platform-msi.c | 202 ++- drivers/dma/Kconfig | 4 drivers/dma/idxd/Makefile | 2 drivers/dma/idxd/cdev.c | 3 drivers/dma/idxd/device.c | 325 ++++- drivers/dma/idxd/dma.c | 9 drivers/dma/idxd/idxd.h | 55 + drivers/dma/idxd/init.c | 81 + drivers/dma/idxd/irq.c | 6 drivers/dma/idxd/mdev.c | 1727 ++++++++++++++++++++++++ drivers/dma/idxd/mdev.h | 105 + drivers/dma/idxd/registers.h | 10 drivers/dma/idxd/submit.c | 31 drivers/dma/idxd/sysfs.c | 199 ++- drivers/dma/idxd/vdev.c | 603 ++++++++ drivers/dma/idxd/vdev.h | 43 + drivers/dma/mv_xor_v2.c | 6 drivers/dma/qcom/hidma.c | 6 drivers/iommu/arm-smmu-v3.c | 6 drivers/iommu/intel-iommu.c | 2 drivers/iommu/intel_irq_remapping.c | 31 drivers/irqchip/irq-mbigen.c | 8 drivers/irqchip/irq-mvebu-icu.c | 6 drivers/mailbox/bcm-flexrm-mailbox.c | 6 drivers/perf/arm_smmuv3_pmu.c | 6 drivers/vfio/mdev/mdev_core.c | 22 drivers/vfio/mdev/mdev_private.h | 2 drivers/vfio/vfio_iommu_type1.c | 52 + include/linux/device.h | 3 include/linux/intel-iommu.h | 3 include/linux/list.h | 36 + include/linux/mdev.h | 13 include/linux/msi.h | 93 + kernel/irq/msi.c | 43 - 42 files changed, 4009 insertions(+), 154 deletions(-) create mode 100644 Documentation/ims-howto.rst create mode 100644 drivers/base/ims-msi.c create mode 100644 drivers/dma/idxd/mdev.c create mode 100644 drivers/dma/idxd/mdev.h create mode 100644 drivers/dma/idxd/vdev.c create mode 100644 drivers/dma/idxd/vdev.h --
