> > > +.. _device_tree_usage: > > > + > > > +Device Tree Usage > > > +----------------- > > > + > > > +The xsabin file stores metadata which advertise HW subsystems present in a > > > +partition. The metadata is stored in device tree format with a well defined > > > +schema. XRT management driver uses this information to bind *xrt_drivers* to > > > +the subsystem instantiations. The xrt_drivers are found in **xrt-lib.ko** kernel > > > +module. > > I'm still catching up the patchset from the very beginning, and just > > finished the Documentation part. So far, I see the DT usage concern > > which may impact the architecure a lot, so I should raise it ASAP. > > > > The concern raised by the DT maintainer: > > https://lore.kernel.org/linux-fpga/CAL_JsqLod6FBGFhu7WXtMrB_z7wj8-up0EetM1QS9M3gjm8d7Q@xxxxxxxxxxxxxx/ > > > > First of all, directly parsing FDT in device drivers is not a normal usage of DT > > in linux. It is out of the current DT usage model. So it should be agreed by DT > > maintainers. > Thanks for reviewing XRT document and providing feedback. > Here is the reply from Sonal for Rob’s question: > https://lore.kernel.org/linux-fpga/BY5PR02MB62604B87C66A1AD139A6F153BBF40@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/ > Overall, libfdt is used by XRT driver to parse the metadata which comes with > an Alveo board. > When XRT driver discovers an Alveo board, it will read a fdt blob from board > firmware file resident on the host. > By parsing the fdt blob, XRT driver gets information about this Alveo board, > such as version, uuid, IPs exposed to PCI BAR, interrupt binding etc. > So libfdt is used simply as Alveo metadata parser here. XRT drivers do not > interact with system wide DT or present the Alveo device tree to host. For > many systems like x86_64, system wide DT is not present but libfdt parsing > services will still be needed. Yes, I understand the use case. My concern is, directly parsing an isolated FDT in device driver and populate sub devices, skipping the unflattening, this is a new working model of device tree usage, but for the same purpose as the existing one. So I really need the confirmation of DT maintainers. > > > > Current FPGA framework modifies kernel's live tree by DT overlay, when FPGA is > > dynamically reprogrammed and new HW devices appear. See > > Documentation/devicetree/bindings/fpga/fpga-region.txt. > > > > Then something less important: > > > > 1. The bindings should be documented in Documentation/devicetree/bindings/. > > 2. Are all the example DT usage conform to the exsiting bindings? I > > didn't go through all device classes, but remember like the > > interrupt-controller should have a "interrupt-controller" property, and > > the PCI properties are also different from PCI bindings. > > The fdt properties are defined for Alveo firmware files. XRT driver is the > only consumer of this data. I am wondering if Personally I don't like the idea of a different binding definition set, even if the new device tree usage is accepted. We all use device tree for device enumeration, so why the different definitions. Thanks, Yilun > Documentation/devicetree/bindings is the right place for Alveo/XRT private > format or should it be documented as part of XRT driver documentation? > Looking for guidance here. > > > Thanks, > > Lizhi > > > > > Thanks, > > Yilun > > > > > + > > > +Logic UUID > > > +^^^^^^^^^^ > > > +A partition is identified uniquely through ``logic_uuid`` property:: > > > + > > > + /dts-v1/; > > > + / { > > > + logic_uuid = "0123456789abcdef0123456789abcdef"; > > > + ... > > > + } > > > + > > > +Schema Version > > > +^^^^^^^^^^^^^^ > > > +Schema version is defined through the ``schema_version`` node. It contains > > > +``major`` and ``minor`` properties as below:: > > > + > > > + /dts-v1/; > > > + / { > > > + schema_version { > > > + major = <0x01>; > > > + minor = <0x00>; > > > + }; > > > + ... > > > + } > > > + > > > +.. _partition_uuids: > > > + > > > +Partition UUIDs > > > +^^^^^^^^^^^^^^^ > > > +Each partition may have parent and child UUIDs. These UUIDs are > > > +defined by ``interfaces`` node and ``interface_uuid`` property:: > > > + > > > + /dts-v1/; > > > + / { > > > + interfaces { > > > + @0 { > > > + interface_uuid = "0123456789abcdef0123456789abcdef"; > > > + }; > > > + @1 { > > > + interface_uuid = "fedcba9876543210fedcba9876543210"; > > > + }; > > > + ... > > > + }; > > > + ... > > > + } > > > + > > > + > > > +Subsystem Instantiations > > > +^^^^^^^^^^^^^^^^^^^^^^^^ > > > +Subsystem instantiations are captured as children of ``addressable_endpoints`` > > > +node:: > > > + > > > + /dts-v1/; > > > + / { > > > + addressable_endpoints { > > > + abc { > > > + ... > > > + }; > > > + def { > > > + ... > > > + }; > > > + ... > > > + } > > > + } > > > + > > > +Subnode 'abc' and 'def' are the name of subsystem nodes > > > + > > > +Subsystem Node > > > +^^^^^^^^^^^^^^ > > > +Each subsystem node and its properties define a hardware instance:: > > > + > > > + > > > + addressable_endpoints { > > > + abc { > > > + reg = <0x00 0x1f05000 0x00 0x1000>> > > > + pcie_physical_function = <0x0>; > > > + pcie_bar_mapping = <0x2>; > > > + compatible = "abc def"; > > > + interrupts = <0x09 0x0c>; > > > + firmware { > > > + firmware_product_name = "abc" > > > + firmware_branch_name = "def" > > > + firmware_version_major = <1> > > > + firmware_version_minor = <2> > > > + }; > > > + } > > > + ... > > > + } > > > + > > > +:reg: > > > + Property defines an address range. `<0x00 0x1f05000 0x00 0x1000>` indicates > > > + *0x00 0x1f05000* as BAR offset and *0x00 0x1000* as address length. > > > +:pcie_physical_function: > > > + Property specifies which PCIe physical function the subsystem node resides. > > > + `<0x0>` implies physical function 0. > > > +:pcie_bar_mapping: > > > + Property specifies which PCIe BAR the subsystem node resides. `<0x2>` implies > > > + BAR 2. A value of 0 means the property is not defined. > > > +:compatible: > > > + Property is a list of strings. The first string in the list specifies the exact > > > + subsystem node. The following strings represent other devices that the device > > > + is compatible with. > > > +:interrupts: > > > + Property specifies start and end interrupts for this subsystem node. > > > + `<0x09 0x0c>` implies interrupts 9 to 13 are used by this subsystem. > > > +:firmware: > > > + Subnode defines the firmware required by this subsystem node. > > > + > > > +Alveo U50 Platform Example > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > +:: > > > + > > > + /dts-v1/; > > > + > > > + /{ > > > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b"; > > > + > > > + schema_version { > > > + major = <0x01>; > > > + minor = <0x00>; > > > + }; > > > + > > > + interfaces { > > > + > > > + @0 { > > > + interface_uuid = "862c7020a250293e32036f19956669e5"; > > > + }; > > > + }; > > > + > > > + addressable_endpoints { > > > + > > > + ep_blp_rom_00 { > > > + reg = <0x00 0x1f04000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; > > > + }; > > > + > > > + ep_card_flash_program_00 { > > > + reg = <0x00 0x1f06000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi"; > > > + interrupts = <0x03 0x03>; > > > + }; > > > + > > > + ep_cmc_firmware_mem_00 { > > > + reg = <0x00 0x1e20000 0x00 0x20000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; > > > + > > > + firmware { > > > + firmware_product_name = "cmc"; > > > + firmware_branch_name = "u50"; > > > + firmware_version_major = <0x01>; > > > + firmware_version_minor = <0x00>; > > > + }; > > > + }; > > > + > > > + ep_cmc_intc_00 { > > > + reg = <0x00 0x1e03000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc"; > > > + interrupts = <0x04 0x04>; > > > + }; > > > + > > > + ep_cmc_mutex_00 { > > > + reg = <0x00 0x1e02000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_cmc_regmap_00 { > > > + reg = <0x00 0x1e08000 0x00 0x2000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; > > > + > > > + firmware { > > > + firmware_product_name = "sc-fw"; > > > + firmware_branch_name = "u50"; > > > + firmware_version_major = <0x05>; > > > + }; > > > + }; > > > + > > > + ep_cmc_reset_00 { > > > + reg = <0x00 0x1e01000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_ddr_mem_calib_00 { > > > + reg = <0x00 0x63000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_debug_bscan_mgmt_00 { > > > + reg = <0x00 0x1e90000 0x00 0x10000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge"; > > > + }; > > > + > > > + ep_ert_base_address_00 { > > > + reg = <0x00 0x21000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_ert_command_queue_mgmt_00 { > > > + reg = <0x00 0x40000 0x00 0x10000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue"; > > > + }; > > > + > > > + ep_ert_command_queue_user_00 { > > > + reg = <0x00 0x40000 0x00 0x10000>; > > > + pcie_physical_function = <0x01>; > > > + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue"; > > > + }; > > > + > > > + ep_ert_firmware_mem_00 { > > > + reg = <0x00 0x30000 0x00 0x8000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; > > > + > > > + firmware { > > > + firmware_product_name = "ert"; > > > + firmware_branch_name = "v20"; > > > + firmware_version_major = <0x01>; > > > + }; > > > + }; > > > + > > > + ep_ert_intc_00 { > > > + reg = <0x00 0x23000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc"; > > > + interrupts = <0x05 0x05>; > > > + }; > > > + > > > + ep_ert_reset_00 { > > > + reg = <0x00 0x22000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_ert_sched_00 { > > > + reg = <0x00 0x50000 0x00 0x1000>; > > > + pcie_physical_function = <0x01>; > > > + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched"; > > > + interrupts = <0x09 0x0c>; > > > + }; > > > + > > > + ep_fpga_configuration_00 { > > > + reg = <0x00 0x1e88000 0x00 0x8000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap"; > > > + interrupts = <0x02 0x02>; > > > + }; > > > + > > > + ep_icap_reset_00 { > > > + reg = <0x00 0x1f07000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_msix_00 { > > > + reg = <0x00 0x00 0x00 0x20000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix"; > > > + pcie_bar_mapping = <0x02>; > > > + }; > > > + > > > + ep_pcie_link_mon_00 { > > > + reg = <0x00 0x1f05000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_pr_isolate_plp_00 { > > > + reg = <0x00 0x1f01000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_pr_isolate_ulp_00 { > > > + reg = <0x00 0x1000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio"; > > > + }; > > > + > > > + ep_uuid_rom_00 { > > > + reg = <0x00 0x64000 0x00 0x1000>; > > > + pcie_physical_function = <0x00>; > > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl"; > > > + }; > > > + > > > + ep_xdma_00 { > > > + reg = <0x00 0x00 0x00 0x10000>; > > > + pcie_physical_function = <0x01>; > > > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma"; > > > + pcie_bar_mapping = <0x02>; > > > + }; > > > + }; > > > + > > > + } > > > + > > > + > > > + > > > +Deployment Models > > > +================= > > > + > > > +Baremetal > > > +--------- > > > + > > > +In bare-metal deployments, both MPF and UPF are visible and accessible. The > > > +xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and > > > +available to system administrator. The full stack is illustrated below:: > > > + > > > + HOST > > > + > > > + [XRT-MGMT] [XRT-USER] > > > + | | > > > + | | > > > + +-----+ +-----+ > > > + | MPF | | UPF | > > > + | | | | > > > + | PF0 | | PF1 | > > > + +--+--+ +--+--+ > > > + ......... ^................. ^.......... > > > + | | > > > + | PCIe DEVICE | > > > + | | > > > + +--+------------------+--+ > > > + | SHELL | > > > + | | > > > + +------------------------+ > > > + | USER | > > > + | | > > > + | | > > > + | | > > > + | | > > > + +------------------------+ > > > + > > > + > > > + > > > +Virtualized > > > +----------- > > > + > > > +In virtualized deployments, the privileged MPF is assigned to the host but the > > > +unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt > > > +driver in host binds to MPF. The xrt-mgmt driver operations are privileged and > > > +only accessible to the MPF. The full stack is illustrated below:: > > > + > > > + > > > + .............. > > > + HOST . VM . > > > + . . > > > + [XRT-MGMT] . [XRT-USER] . > > > + | . | . > > > + | . | . > > > + +-----+ . +-----+ . > > > + | MPF | . | UPF | . > > > + | | . | | . > > > + | PF0 | . | PF1 | . > > > + +--+--+ . +--+--+ . > > > + ......... ^................. ^.......... > > > + | | > > > + | PCIe DEVICE | > > > + | | > > > + +--+------------------+--+ > > > + | SHELL | > > > + | | > > > + +------------------------+ > > > + | USER | > > > + | | > > > + | | > > > + | | > > > + | | > > > + +------------------------+ > > > + > > > + > > > + > > > + > > > + > > > +Platform Security Considerations > > > +================================ > > > + > > > +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_ > > > +discusses the deployment options and security implications in great detail. > > > diff --git a/MAINTAINERS b/MAINTAINERS > > > index 056966c9aac9..beeaf0257364 100644 > > > --- a/MAINTAINERS > > > +++ b/MAINTAINERS > > > @@ -7274,6 +7274,17 @@ F: Documentation/fpga/ > > > F: drivers/fpga/ > > > F: include/linux/fpga/ > > > > > > +FPGA XRT DRIVERS > > > +M: Lizhi Hou <lizhi.hou@xxxxxxxxxx> > > > +R: Max Zhen <max.zhen@xxxxxxxxxx> > > > +R: Sonal Santan <sonal.santan@xxxxxxxxxx> > > > +L: linux-fpga@xxxxxxxxxxxxxxx > > > +S: Supported > > > +W: https://github.com/Xilinx/XRT > > > +F: Documentation/fpga/xrt.rst > > > +F: drivers/fpga/xrt/ > > > +F: include/uapi/linux/xrt/ > > > + > > > FPU EMULATOR > > > M: Bill Metzenthen <billm@xxxxxxxxxxxxx> > > > S: Maintained > > > -- > > > 2.27.0
