Devices in FPGA can be added/modified dynamically on run-time.
This patch series introduces AMD CDX bus, which provides a
mechanism to discover/rescan FPGA devices on run-time. These
devices are memory mapped on system bus for embedded CPUs, and
added as CDX devices in Linux framework.
This RFC:
- Intrduces the CDX bus controller and CDX devices.
- Adds rescan and reset support for the CDX bus.
- Add support for reset for CDX devices.
- Support for CDX bus in arm-smmu-v3 driver
- Support for CDX-MSI domain.
- Vfio-cdx driver support for CDX devices.
Please NOTE: This RFC change does not support the CDX bus firmware
interface as it is under development, and this series aims to get
an early feedback from the community. Firmware interaction are
stubbed as MCDI APIs which is a protocol used by AMD to interact
with Firmware.
Changes in v2:
- introduce CDX bus infrastructure
- fixed code for making compatible visible for devices
having the 'compatible' property only (Greg K-H)
- moved CDX-MSI domain as part of CDX bus infrastructure.
previously it was part of irqchip (Marc Zynger).
- fixed few prints (Greg K-H)
- support rescan and reset of CDX bus
- add VFIO reset module for CDX bus based devices
Changes in v3:
- Move CDX bus as a new bus type in kernel rather than
using the platform devices (Greg K-H, Marc Zynger)
- Correspondingly update ARM SMMU v3
- Add support for vfio-cdx driver
- Updated device tree yaml with correct binding information
(Krzysztof Kozlowski)
- remove 'compatible' sysfs platform patch which was requried
for CDX devices exposed as platform devices
Following text provides a basic overview of CDX bus Architecture.
Contents summary
- CDX overview
- CDX Linux driver architecture overview
- Bus driver
- VFIO driver
CDX Overview
------------
CDX is a Hardware Architecture designed for AMD FPGA devices. It
consists of sophisticated mechanism for interaction between FPGA,
Firmware and the APUs (Application CPUs).
Firmware resides on RPU (Realtime CPUs) which interacts with
the FPGA program manager and the APUs. The RPU provides memory-mapped
interface (RPU if) which is used to communicate with APUs.
The diagram below shows an overview of the CDX architecture:
+--------------------------------------+
| Application CPUs (APU) |
| |
| CDX device drivers|
| Linux OS | |
| CDX bus |
| | |
+-----------------------------|--------+
| (discover, config,
| reset, rescan)
|
+------------------------| RPU if |----+
| | |
| V |
| Realtime CPUs (RPU) |
| |
+--------------------------------------+
|
+---------------------|----------------+
| FPGA | |
| +-----------------------+ |
| | | | |
| +-------+ +-------+ +-------+ |
| | dev 1 | | dev 2 | | dev 3 | |
| +-------+ +-------+ +-------+ |
+--------------------------------------+
The RPU firmware extracts the device information from the loaded FPGA
image and implements a mechanism that allows the APU drivers to
enumerate such devices (device personality and resource details) via
a dedicated communication channel. RPU mediates operations such as
discover, reset and rescan of the FPGA devices for the APU. This is
done using memory mapped interface provided by the RPU to APU.
Nipun Gupta (7):
dt-bindings: bus: add CDX bus device tree bindings
bus/cdx: add the cdx bus driver
iommu/arm-smmu-v3: support ops registration for CDX bus
bus/cdx: add cdx-MSI domain with gic-its domain as parent
bus/cdx: add bus and device attributes
vfio/cdx: add support for CDX bus
vfio/cdx: add interrupt support
Documentation/ABI/testing/sysfs-bus-cdx | 54 ++
.../devicetree/bindings/bus/xlnx,cdx.yaml | 75 +++
MAINTAINERS | 9 +
drivers/bus/Kconfig | 1 +
drivers/bus/Makefile | 3 +
drivers/bus/cdx/Kconfig | 7 +
drivers/bus/cdx/Makefile | 3 +
drivers/bus/cdx/cdx.c | 603 ++++++++++++++++++
drivers/bus/cdx/cdx.h | 53 ++
drivers/bus/cdx/cdx_msi.c | 236 +++++++
drivers/bus/cdx/mcdi_stubs.c | 61 ++
drivers/bus/cdx/mcdi_stubs.h | 87 +++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 16 +-
drivers/vfio/Makefile | 1 +
drivers/vfio/cdx/Kconfig | 10 +
drivers/vfio/cdx/Makefile | 4 +
drivers/vfio/cdx/vfio_cdx.c | 337 ++++++++++
drivers/vfio/cdx/vfio_cdx_intr.c | 212 ++++++
drivers/vfio/cdx/vfio_cdx_private.h | 50 ++
include/linux/cdx/cdx_bus.h | 120 ++++
include/linux/mod_devicetable.h | 13 +
scripts/mod/devicetable-offsets.c | 4 +
scripts/mod/file2alias.c | 12 +
23 files changed, 1969 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-bus-cdx
create mode 100644 Documentation/devicetree/bindings/bus/xlnx,cdx.yaml
create mode 100644 drivers/bus/cdx/Kconfig
create mode 100644 drivers/bus/cdx/Makefile
create mode 100644 drivers/bus/cdx/cdx.c
create mode 100644 drivers/bus/cdx/cdx.h
create mode 100644 drivers/bus/cdx/cdx_msi.c
create mode 100644 drivers/bus/cdx/mcdi_stubs.c
create mode 100644 drivers/bus/cdx/mcdi_stubs.h
create mode 100644 drivers/vfio/cdx/Kconfig
create mode 100644 drivers/vfio/cdx/Makefile
create mode 100644 drivers/vfio/cdx/vfio_cdx.c
create mode 100644 drivers/vfio/cdx/vfio_cdx_intr.c
create mode 100644 drivers/vfio/cdx/vfio_cdx_private.h
create mode 100644 include/linux/cdx/cdx_bus.h
--
2.25.1
