On Mon, Nov 27, 2017 at 12:42 AM, Wu Hao <hao.wu@xxxxxxxxx> wrote: > Add a document for Intel FPGA driver overview. > > Signed-off-by: Enno Luebbers <enno.luebbers@xxxxxxxxx> > Signed-off-by: Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx> > Signed-off-by: Wu Hao <hao.wu@xxxxxxxxx> > ---- > v2: added FME fpga-mgr/bridge/region platform driver to driver organization. > updated open discussion per current implementation. > fixed some typos. > v3: use FPGA base region as container device instead of fpga-dev class. > split common enumeration code from pcie driver to functions exposed by > device feature list framework. > update FME performance reporting which supports both integrated (iperf/) > and discrete (dperf/) FPGA solutions. > --- > Documentation/fpga/intel-fpga.txt | 261 ++++++++++++++++++++++++++++++++++++++ > 1 file changed, 261 insertions(+) > create mode 100644 Documentation/fpga/intel-fpga.txt > > diff --git a/Documentation/fpga/intel-fpga.txt b/Documentation/fpga/intel-fpga.txt > new file mode 100644 > index 0000000..0754733 > --- /dev/null > +++ b/Documentation/fpga/intel-fpga.txt > @@ -0,0 +1,261 @@ > +=============================================================================== > + Intel FPGA driver Overview This doesn't look Intel specific to me. This could all be 'DFL FPGA Framework' > +------------------------------------------------------------------------------- > + Enno Luebbers <enno.luebbers@xxxxxxxxx> > + Xiao Guangrong <guangrong.xiao@xxxxxxxxxxxxxxx> > + Wu Hao <hao.wu@xxxxxxxxx> > + > +The Intel FPGA driver provides interfaces for userspace applications to > +configure, enumerate, open, and access FPGA accelerators on platforms equipped > +with Intel(R) FPGA PCIe based solutions and enables system level management > +functions such as FPGA reconfiguration, power management, and virtualization. > + > +HW Architecture > +=============== > +From the OS's point of view, the FPGA hardware appears as a regular PCIe device. > +The FPGA device memory is organized using a predefined data structure (Device > +Feature List). Features supported by the particular FPGA device are exposed > +through these data structures, as illustrated below: > + > + +-------------------------------+ +-------------+ > + | PF | | VF | > + +-------------------------------+ +-------------+ > + ^ ^ ^ ^ > + | | | | > ++-----|------------|---------|--------------|-------+ > +| | | | | | > +| +-----+ +-------+ +-------+ +-------+ | > +| | FME | | Port0 | | Port1 | | Port2 | | > +| +-----+ +-------+ +-------+ +-------+ | > +| ^ ^ ^ | > +| | | | | > +| +-------+ +------+ +-------+ | > +| | AFU | | AFU | | AFU | | > +| +-------+ +------+ +-------+ | > +| | > +| FPGA PCIe Device | > ++---------------------------------------------------+ > + > +The driver supports PCIe SR-IOV to create virtual functions (VFs) which can be > +used to assign individual accelerators to virtual machines. > + > +FME (FPGA Management Engine) > +============================ > +The FPGA Management Engine performs power and thermal management, error > +reporting, reconfiguration, performance reporting for integrated and discrete > +solution, and other infrastructure functions. Each FPGA has one FME, which is > +always accessed through the physical function (PF). > + > +User-space applications can acquire exclusive access to the FME using open(), > +and release it using close(). > + > +The following functions are exposed through ioctls: > + > + Get driver API version (FPGA_GET_API_VERSION) > + Check for extensions (FPGA_CHECK_EXTENSION) > + Assign port to PF (FPGA_FME_PORT_ASSIGN) > + Release port from PF (FPGA_FME_PORT_RELEASE) > + Program bitstream (FPGA_FME_PORT_PR) > + > +More functions are exposed through sysfs > +(/sys/class/fpga_region/regionX/fpga-dfl-fme.n/): I see that /sys/class/fpga/* has changed to /sys/class/fpga_region/* now as requested (thanks!). It looks like it ended up being pretty straightforward (so far, just diffing this doc with the previous v2). > + > + Read bitstream ID (bitstream_id) > + Read bitstream metadata (bitstream_metadata) > + Read number of ports (ports_num) > + Read socket ID (socket_id) > + Read performance counters (iperf/ and dperf/) > + Power management (power_mgmt/) > + Thermal management (thermal_mgmt/) > + Error reporting (errors/) > + > +PORT > +==== > +A port represents the interface between the static FPGA fabric (the "blue > +bitstream") and a partially reconfigurable region containing an AFU (the "green > +bitstream"). It controls the communication from SW to the accelerator and > +exposes features such as reset and debug. > + > +A PCIe device may have several ports and each port can be released from PF by > +FPGA_FME_PORT_RELEASE ioctl on FME, and exposed through a VF via PCIe sriov > +sysfs interface. > + > +AFU > +=== > +An AFU is attached to a port and exposes a 256k MMIO region to be used for > +accelerator-specific control registers. > + > +User-space applications can acquire exclusive access to an AFU attached to a > +port by using open() on the port device node, and release it using close(). > + > +The following functions are exposed through ioctls: > + > + Get driver API version (FPGA_GET_API_VERSION) > + Check for extensions (FPGA_CHECK_EXTENSION) > + Get port info (FPGA_PORT_GET_INFO) > + Get MMIO region info (FPGA_PORT_GET_REGION_INFO) > + Map DMA buffer (FPGA_PORT_DMA_MAP) > + Unmap DMA buffer (FPGA_PORT_DMA_UNMAP) > + Reset AFU (FPGA_PORT_RESET) > + Enable UMsg (FPGA_PORT_UMSG_ENABLE) > + Disable UMsg (FPGA_PORT_UMSG_DISABLE) > + Set UMsg mode (FPGA_PORT_UMSG_SET_MODE) > + Set UMsg base address (FPGA_PORT_UMSG_SET_BASE_ADDR) > + > +User-space applications can also mmap() accelerator MMIO regions. > + > +More functions are exposed through sysfs: > +(/sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/): > + > + Read Accelerator GUID (afu_id) > + Error reporting (errors/) > + > +Partial Reconfiguration > +======================= > +As mentioned above, accelerators can be reconfigured through partial > +reconfiguration of a green bitstream file (GBS). The green bitstream must have > +been generated for the exact blue bitstream and targeted reconfigurable region > +(port) of the FPGA; otherwise, the reconfiguration operation will fail and > +possibly cause system instability. This compatibility can be checked by > +comparing the interface ID noted in the GBS header against the interface ID > +exposed by the FME through sysfs (see above). This check is usually done by > +user-space before calling the reconfiguration IOCTL. > + > +FPGA virtualization > +=================== > +To enable accessing an accelerator from applications running in a VM, the > +respective AFU's port needs to be assigned to a VF using the following steps: > + > + a) The PF owns all AFU ports by default. Any port that needs to be reassigned > + to a VF must first be released through the FPGA_FME_PORT_RELEASE ioctl on the > + FME device. > + > + b) Once N ports are released from PF, then user can use command below to > + enable SRIOV and VFs. Each VF owns only one Port with AFU. > + > + echo N > $PCI_DEVICE_PATH/sriov_numvfs > + > + c) Pass through the VFs to VMs > + > + d) The AFU under VF is accessible from applications in VM (using the same > + driver inside the VF). > + > +Note that an FME can't be assigned to a VF, thus PR and other management > +functions are only available via the PF. > + > + > +Driver organization > +=================== > + > + +-------++------++------+ | > + | FME || FME || FME | | > + | FPGA || FPGA || FPGA | | > + |Manager||Bridge||Region| | > + +-------++------++------+ | > + +-----------------------+ +--------+ | +--------+ > + | FME | | AFU | | | AFU | > + | Module | | Module | | | Module | > + +-----------------------+ +--------+ | +--------+ > + +-----------------------+ | +-----------------------+ > + | FPGA Container Device | | | FPGA Container Device | > + | (FPGA Base Region) | | | (FPGA Base Region) | > + +-----------------------+ | +-----------------------+ > + +------------------+ | +------------------+ > + | FPGA PCIE Module | | Virtual | FPGA PCIE Module | > + +------------------+ Host | Machine +------------------+ > + -------------------------------------- | ------------------------------ > + +---------------+ | +---------------+ > + | PCI PF Device | | | PCI VF Device | > + +---------------+ | +---------------+ > + > +The FPGA devices appear as regular PCIe devices; thus, the FPGA PCIe device > +driver is always loaded first once a FPGA PCIE PF or VF device is detected. This > +driver plays an infrastructural role in the driver architecture. It: > + > + a) locates the Device Feature Lists in PCIE device BAR memory, handles > + them and related resources to common interfaces from DFL framework > + for enumeration. > + b) supports SRIOV. > + > +The feature device infrastructure provides common interfaces to create container > +device (FPGA base region), discover feature devices and their sub features from > +the given Device Feature Lists, and create platform devices for feature devices > +with related resources under the container device. It also abstracts operations > +for sub features and exposes common interfaces to feature device drivers. > + > +The FPGA Management Engine (FME) driver is a platform driver which is loaded > +automatically after FME platform device creation from the PCIE driver. It > +provides the key features for FPGA management, including: > + > + a) Power and thermal management, error reporting, performance reporting > + and other infrastructure functions. Users can access these functions > + via sysfs interfaces exposed by FME driver. > + b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA > + bridges and FPGA regions during PR sub feature initialization; Once > + it receives an FPGA_FME_PORT_PR ioctl from user, it invokes the > + common interface function from FPGA Region to complete the partial > + reconfiguration of the bitstream to the given port. > + c) Port management for virtualization. The FME driver introduces two > + ioctls, FPGA_FME_PORT_RELEASE (releases given port from PF) and > + FPGA_FME_PORT_ASSIGN (assigns the port back to PF). Once the port is > + released from the PF, it can be assigned to the VF through the SRIOV > + interfaces provided by PCIE driver. (Refer to "FPGA virtualization" > + for more details). > + > +Similar to the the FME driver, the FPGA Accelerated Function Unit (AFU) driver > +is probed once the AFU platform device is created. The main function of this > +module is to provide an interface for userspace applications to access the > +individual accelerators, including basic reset control on port, AFU MMIO region > +export, dma buffer mapping service, UMsg notification, and remote debug > +functions (see above). > + > + > +Device enumeration > +================== > +This section introduces how applications enumerate the fpga device from > +the sysfs hierarchy under /sys/class/fpga_region. > + > +In the example below, two Intel(R) FPGA devices are installed in the host. Each > +fpga device has one FME and two ports (AFUs). > + > +FPGA regions are created under /sys/class/fpga_region/ > + > + /sys/class/fpga_region/region0 > + /sys/class/fpga_region/region1 > + /sys/class/fpga_region/region2 > + ... > + > +Application needs to search each regionX folder, if feature device is found, > +(e.g "fpga-dfl-port.n" or "fpga-dfl-fme.m" is found), then it's the base > +fpga region which represents the FPGA device. > + > +Each base region has one FME and two ports (AFUs) as child devices: > + > + /sys/class/fpga_region/region0/fpga-dfl-fme.0 > + /sys/class/fpga_region/region0/fpga-dfl-port.0 > + /sys/class/fpga_region/region0/fpga-dfl-port.1 > + ... > + > + /sys/class/fpga_region/region3/fpga-dfl-fme.1 > + /sys/class/fpga_region/region3/fpga-dfl-port.2 > + /sys/class/fpga_region/region3/fpga-dfl-port.3 > + ... > + > +In general, the FME/AFU sysfs interfaces are named as follows: > + > + /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/ > + /sys/class/fpga_region/<regionX>/<fpga-dfl-port.m>/ > + > +with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all > +ports. > + > +The device nodes used for ioctl() or mmap() can be referenced through: > + > + /sys/class/fpga_region/<regionX>/<fpga-dfl-fme.n>/dev > + /sys/class/fpga_region/<regionX>/<fpga-dfl-port.n>/dev > + > +Open discussion > +=============== > +FME driver exports one ioctl (FPGA_FME_PORT_PR) for partial reconfiguration to > +user now. In the future, if unified user interfaces for reconfiguration are > +added, FME driver should switch to them from ioctl interface. > -- > 1.8.3.1 > -- To unsubscribe from this list: send the line "unsubscribe linux-api" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
