Add specification for the *PCI NTB* function device. The endpoint function driver and the host PCI driver should be created based on this specification. Signed-off-by: Kishon Vijay Abraham I <kishon@xxxxxx> --- Documentation/PCI/endpoint/pci-test-ntb.txt | 315 ++++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt diff --git a/Documentation/PCI/endpoint/pci-test-ntb.txt b/Documentation/PCI/endpoint/pci-test-ntb.txt new file mode 100644 index 000000000000..c8bfe9dbfd8b --- /dev/null +++ b/Documentation/PCI/endpoint/pci-test-ntb.txt @@ -0,0 +1,315 @@ + PCI NTB FUNCTION + Kishon Vijay Abraham I <kishon@xxxxxx> + +PCI NTB Function allows two different systems (or hosts) to communicate +with each other by configurig the endpoint instances in such a way that +transactions from one system is routed to the other system. + +In the below diagram, PCI NTB function configures the SoC with multiple +PCIe Endpoint (EP) instances in such a way that transaction from one EP +controller is routed to the other EP controller. Once PCI NTB function +configures the SoC with multiple EP instances, HOST1 and HOST2 can +communicate with each other using SoC as a bridge. + + +-------------+ +-------------+ + | | | | + | HOST1 | | HOST2 | + | | | | + +------^------+ +------^------+ + | | + | | ++---------|-------------------------------------------------|---------+ +| +------v------+ +------v------+ | +| | | | | | +| | EP | | EP | | +| | CONTROLLER1 | | CONTROLLER2 | | +| | <-----------------------------------> | | +| | | | | | +| | | | | | +| | | SoC With Multiple EP Instances | | | +| | | (Configured using NTB Function) | | | +| +-------------+ +-------------+ | ++---------------------------------------------------------------------+ + +Constructs used for Implementing NTB: + + *) Config Region + *) Self Scratchpad Registers + *) Peer Scratchpad Registers + *) Doorbell Registers + *) Memory Window + +Modeling Constructs: + + There are 5 or more distinct regions (config, self scratchpad, peer +scratchpad, doorbell, one or more memory windows) to be modeled to achieve +NTB functionality. Atleast one memory window is required while more than +one is permitted. All these regions should be mapped to BAR for hosts to +access these regions. + +If one 32-bit BAR is allocated for each of these regions, the scheme would +look like + BAR0 -> Config Region + BAR1 -> Self Scratchpad + BAR2 -> Peer Scratchpad + BAR3 -> Doorbell + BAR4 -> Memory Window 1 + BAR5 -> Memory Window 2 + +However if we allocate a separate BAR for each of the region, there would not +be enough BARs for all the regions in a platform that supports only 64-bit +BAR. + +In order to be be supported by most of the platforms, the regions should be +packed and mapped to BARs in a way that provides NTB functionality and +also making sure the hosts doesn't access any region that it is not supposed +to. + +The following scheme is used in EPF NTB Function + + BAR0 -> Config Region + Self Scratchpad + BAR1 -> Peer Scratchpad + BAR2 -> Doorbell + Memory Window 1 + BAR3 -> Memory Window 2 + BAR4 -> Memory Window 3 + BAR4 -> Memory Window 4 + +With this scheme, for the basic NTB functionality 3 BARs should be sufficient. + +Modeling Config/Scratchpad Region: + ++-----------------+------->+------------------+ +-----------------+ +| BAR0 | | CONFIG REGION | | BAR0 | ++-----------------+----+ +------------------+<-------+-----------------+ +| BAR1 | | |SCRATCHPAD REGION | | BAR1 | ++-----------------+ +-->+------------------+<-------+-----------------+ +| BAR2 | Local Memory | BAR2 | ++-----------------+ +-----------------+ +| BAR3 | | BAR3 | ++-----------------+ +-----------------+ +| BAR4 | | BAR4 | ++-----------------+ +-----------------+ +| BAR5 | | BAR5 | ++-----------------+ +-----------------+ + EP CONTROLLER 1 EP CONTROLLER 2 + +Above diagram shows Config region + Scratchpad region for HOST1 (connected to +EP controller 1) allocated in local memory. The HOST1 can access the config +region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. +The peer host (HOST2 connected to EP controller 2) can also access this +scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This +diagram shows the case where Config region and Scratchpad region is allocated +for HOST1, however the same is applicable for HOST2. + +Modeling Doorbell/Memory Window 1: + ++-----------------+ +----->+----------------+-----------+-----------------+ +| BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 | ++-----------------+ | +----------------+ +-----------------+ +| BAR1 | | | Doorbell 2 +---------+ | | ++-----------------+ | +----------------+ | | | +| BAR2 | | | Doorbell 3 +-------+ | +-----------------+ ++-----------------+ | +----------------+ | +-> MSI|X ADDRESS 2 | +| BAR3 | | | Doorbell 4 +-----+ | +-----------------+ ++----------------------+ +----------------+ | | | | +| BAR4 | | | | | +-----------------+ ++----------------------+ | MW1 +---+ | +-->+ MSI|X ADDRESS 3|| +| BAR5 | | | | | | +-----------------+ ++-----------------+ +----->-----------------+ | | | | + EP CONTROLLER 1 | | | | +-----------------+ + | | | +---->+ MSI|X ADDRESS 4 | + +----------------+ | +-----------------+ + EP CONTROLLER 2 | | | + (OB SPACE) | | | + +-------> MW1 | + | | + | | + +-----------------+ + | | + | | + | | + | | + | | + +-----------------+ + PCI Address Space + (Managed by HOST2) + +Above diagram shows how the doorbell and memory window 1 is mapped so that +HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access +buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and +memory window 1 regions are allocated in EP controller 2 outbound (OB) address +space. Allocating and configuring BARs for doorbell and memory window1 +is done during the initialization phase of NTB endpoint function driver. +Mapping from EP controller 2 OB space to PCI address space is done when HOST2 +sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. The commands are explained +below. + +Modeling Optional Memory Windows: + +This is modeled the same was as MW1 but each of the additional memory windows +is mapped to separate BARs. + +Config Region: + +Config Region is a construct that is specific to NTB implemented using NTB +Endpoint Function Driver. The host and endpoint side NTB function driver will +exchange informatio with each other using this region. Config Region has +Control/Status Registers for configuring the Endpoint Controller. Host can +write into this region for configuring the outbound ATU and to indicate the +link status. Endpoint can indicate the status of commands issued be host in +this region. Endpoint can also indicate the scratchpad offset, number of +memory windows to the host using this region. + +The format of Config Region is given below. Each of the fields here are 32 +bits. + + +------------------------+ + | COMMAND | + +------------------------+ + | ARGUMENT | + +------------------------+ + | STATUS | + +------------------------+ + | TOPOLOGY | + +------------------------+ + | ADDRESS (LOWER 32) | + +------------------------+ + | ADDRESS (UPPER 32) | + +------------------------+ + | SIZE | + +------------------------+ + | MEMORY WINDOW1 OFFSET | + +------------------------+ + | NO OF MEMORY WINDOW | + +------------------------+ + | SPAD OFFSET | + +------------------------+ + | SPAD COUNT | + +------------------------+ + | DB ENTRY SIZE | + +------------------------+ + | DB DATA | + +------------------------+ + | : | + +------------------------+ + | : | + +------------------------+ + | DB DATA | + +------------------------+ + + + COMMAND: + + NTB function supports three commands: + + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before + invoking this command, the host should allocate and initialize + MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the + Endpoint). The endpoint on receiving this command will configure + the outbound ATU such that transaction to DB BAR will be routed + to the MSI/MSI-X address programmed by the host. The ARGUMENT + register should be populated with number of DBs to configure (in the + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). + (TODO: Add support for MSI-X). + + CMD_CONFIGURE_MW (0x2): Command to configure memory window. The + host invokes this command after allocating a buffer that can be + accessed by remote host. The allocated address should be programmed + in the ADDRESS register (64 bit), the size should be programmed in + the SIZE register and the memory window index should be programmed + in the ARGUMENT register. The endpoint on receiving this command + will configure the outbound ATU such that trasaction to MW BAR + will be routed to the address provided by the host. + + CMD_LINK_UP (0x3): Command to indicate an NTB application is + bound to the EP device on the host side. Once the endpoint + receives this command from both the hosts, the endpoint will + raise an LINK_UP event to both the hosts to indicate the hosts + can start communicating with each other. + + ARGUMENT: + + The value of this register is based on the commands issued in + command register. See COMMAND section for more information. + + configuring memory window and to indicate the host side NTB application + has initialized. + + TOPOLOGY: + + Set to NTB_TOPO_B2B_USD for Primary interface + Set to NTB_TOPO_B2B_DSD for Secondary interface + + ADDRESS/SIZE: + + Address and Size to be used while configuring the memory window. + See "CMD_CONFIGURE_MW" for more info. + + MEMORY WINDOW1 OFFSET: + + Memory Window 1 and Doorbell registers are packed together in the + same BAR. The initial portion of the region will have doorbell + registers and the latter portion of the region is for memory window 1. + This register will specify the offset of the memory window 1. + + NO OF MEMORY WINDOW: + + Specifies the number of memory windows supported by the NTB device. + + SPAD OFFSET: + + Self scratchpad region and config region are packed together in the + same BAR. The initial portion of the will have config region and + the latter portion of the region is for self scratchpad. This + register will specify the offset of the self scratchpad registers. + + SPAD COUNT: + + Specifies the number of scratchpad registers supported by the NTB + device. + + DB ENTRY SIZE: + + Used to determine the offset within the DB BAR that should be written + in order to raise doorbell. EPF NTB can use either MSI/MSI-X to + ring doorbell (MSI-X support will be added later). MSI uses same + address for all the interrupts and MSI-X can provide different + addresses for different interrupts. The MSI/MSI-X address is provided + by the host and the address it gives is based on the MSI/MSI-X + implementation supported by the host. For instance, ARM platform + using GIC ITS will have same MSI-X address for all the interrupts. + In order to support all the combinations and use the same mechanism + for both MSI and MSI-X, EPF NTB allocates separate region in the + Outbound Address Space for each of the interrupts. This region will + be mapped to the MSI/MSI-X address provided by the host. If a host + provides the same address for all the interrupts, all the regions + will be translated to the same address. If a host provides different + address, the regions will be translated to different address. This + will ensure there is no difference while raising the doorbell. + + DB DATA: + + EPF NTB supports 32 interrupts. So there are 32 DB DATA registers. + This holds the MSI/MSI-X data that has to be written to MSI address + for raising doorbell interrupt. This will be populated by EPF NTB + while invoking CMD_CONFIGURE_MW. + +Scratchpad Registers: + + Each host has it's own register space allocated in the memory of NTB EPC. + They are both readable and writable from both sides of the bridge. They + are used by applications built over NTB and can be used to pass control + and status information between both sides of a device. + + Scratchpad registers has 2 parts + 1) Self Scratchpad: Host's own register space + 2) Peer Scratchpad: Remote host's register space. + +Doorbell Registers: + + Registers using which one host can interrupt the other host. + +Memory Window: + + Actual transfer of data between the two hosts will happen using the + memory window. -- 2.17.1
