On Fri, Jan 22, 2021 at 07:48:52PM +0530, Kishon Vijay Abraham I wrote: > Hi Bjorn, > > On 20/01/21 12:04 am, Bjorn Helgaas wrote: > > On Mon, Jan 04, 2021 at 08:58:53PM +0530, Kishon Vijay Abraham I wrote: > >> Add specification for the *PCI NTB* function device. The endpoint function > >> driver and the host PCI driver should be created based on this > >> specification. > >> > >> Signed-off-by: Kishon Vijay Abraham I <kishon@xxxxxx> > > > > A few typos below if there's opportunity for revisions. > > I'll fix them. Hi Kishon, if you have changes please send them along and I will re-merge the whole series. Thanks, Lorenzo > >> --- > >> Documentation/PCI/endpoint/index.rst | 1 + > >> .../PCI/endpoint/pci-ntb-function.rst | 351 ++++++++++++++++++ > >> 2 files changed, 352 insertions(+) > >> create mode 100644 Documentation/PCI/endpoint/pci-ntb-function.rst > >> > >> diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst > >> index 4ca7439fbfc9..ef6861128506 100644 > >> --- a/Documentation/PCI/endpoint/index.rst > >> +++ b/Documentation/PCI/endpoint/index.rst > >> @@ -11,5 +11,6 @@ PCI Endpoint Framework > >> pci-endpoint-cfs > >> pci-test-function > >> pci-test-howto > >> + pci-ntb-function > >> > >> function/binding/pci-test > >> diff --git a/Documentation/PCI/endpoint/pci-ntb-function.rst b/Documentation/PCI/endpoint/pci-ntb-function.rst > >> new file mode 100644 > >> index 000000000000..a57908be4047 > >> --- /dev/null > >> +++ b/Documentation/PCI/endpoint/pci-ntb-function.rst > >> @@ -0,0 +1,351 @@ > >> +.. SPDX-License-Identifier: GPL-2.0 > >> + > >> +================= > >> +PCI NTB Function > >> +================= > >> + > >> +:Author: Kishon Vijay Abraham I <kishon@xxxxxx> > >> + > >> +PCI Non Transparent Bridges (NTB) allow two host systems to communicate > >> +with each other by exposing each host as a device to the other host. > >> +NTBs typically support the ability to generate interrupts on the remote > >> +machine, expose memory ranges as BARs and perform DMA. They also support > >> +scratchpads which are areas of memory within the NTB that are accessible > >> +from both machines. > >> + > >> +PCI NTB Function allows two different systems (or hosts) to communicate > >> +with each other by configurig the endpoint instances in such a way that > >> +transactions from one system is routed to the other system. > > > > s/is/are/ > > > >> +In the below diagram, PCI NTB function configures the SoC with multiple > >> +PCIe Endpoint (EP) instances in such a way that transaction from one EP > >> +controller is routed to the other EP controller. Once PCI NTB function > > > > s/transaction ... is/transactions ... are/ > > > >> +configures the SoC with multiple EP instances, HOST1 and HOST2 can > >> +communicate with each other using SoC as a bridge. > >> + > >> +.. code-block:: text > >> + > >> + +-------------+ +-------------+ > >> + | | | | > >> + | HOST1 | | HOST2 | > >> + | | | | > >> + +------^------+ +------^------+ > >> + | | > >> + | | > >> + +---------|-------------------------------------------------|---------+ > >> + | +------v------+ +------v------+ | > >> + | | | | | | > >> + | | EP | | EP | | > >> + | | CONTROLLER1 | | CONTROLLER2 | | > >> + | | <-----------------------------------> | | > >> + | | | | | | > >> + | | | | | | > >> + | | | SoC With Multiple EP Instances | | | > >> + | | | (Configured using NTB Function) | | | > >> + | +-------------+ +-------------+ | > >> + +---------------------------------------------------------------------+ > >> + > >> +Constructs used for Implementing NTB > >> +==================================== > >> + > >> + 1) Config Region > >> + 2) Self Scratchpad Registers > >> + 3) Peer Scratchpad Registers > >> + 4) Doorbell Registers > >> + 5) Memory Window > >> + > >> + > >> +Config Region: > >> +-------------- > >> + > >> +Config Region is a construct that is specific to NTB implemented using NTB > >> +Endpoint Function Driver. The host and endpoint side NTB function driver will > >> +exchange information with each other using this region. Config Region has > >> +Control/Status Registers for configuring the Endpoint Controller. Host can > >> +write into this region for configuring the outbound ATU and to indicate the > > > > Expand "ATU" since this is the first mention. > > > >> +link status. Endpoint can indicate the status of commands issued be host in > >> +this region. Endpoint can also indicate the scratchpad offset, number of > >> +memory windows to the host using this region. > > > > s/be host/by host/ > > s/offset, number/offset and number/ > > > >> +The format of Config Region is given below. Each of the fields here are 32 > >> +bits. > > > > s/Each ... are/All ... are/ > > > >> + > >> +.. code-block:: text > >> + > >> + +------------------------+ > >> + | COMMAND | > >> + +------------------------+ > >> + | ARGUMENT | > >> + +------------------------+ > >> + | STATUS | > >> + +------------------------+ > >> + | TOPOLOGY | > >> + +------------------------+ > >> + | ADDRESS (LOWER 32) | > >> + +------------------------+ > >> + | ADDRESS (UPPER 32) | > >> + +------------------------+ > >> + | SIZE | > >> + +------------------------+ > >> + | NO OF MEMORY WINDOW | > >> + +------------------------+ > >> + | MEMORY WINDOW1 OFFSET | > >> + +------------------------+ > >> + | SPAD OFFSET | > >> + +------------------------+ > >> + | SPAD COUNT | > >> + +------------------------+ > >> + | DB ENTRY SIZE | > >> + +------------------------+ > >> + | DB DATA | > >> + +------------------------+ > >> + | : | > >> + +------------------------+ > >> + | : | > >> + +------------------------+ > >> + | DB DATA | > >> + +------------------------+ > >> + > >> + > >> + COMMAND: > >> + > >> + NTB function supports three commands: > >> + > >> + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before > >> + invoking this command, the host should allocate and initialize > >> + MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the > > > > s/i.e/i.e.,/ > > > >> + Endpoint). The endpoint on receiving this command will configure > >> + the outbound ATU such that transaction to DB BAR will be routed > >> + to the MSI/MSI-X address programmed by the host. The ARGUMENT > > > > s/transaction to/transactions to/ > > > > Expand "DB BAR". I assume this refers to "Doorbell BAR" (which itself > > is not defined). How do we know which is the Doorbell BAR? > > right doorbell. That part is explained in the "Modeling Constructs" > section below. > > > > Also, "DB" itself needs to be expanded somehow for uses like below: > > > >> + register should be populated with number of DBs to configure (in the > >> + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). > >> + (TODO: Add support for MSI-X). > >> + > >> + CMD_CONFIGURE_MW (0x2): Command to configure memory window. The > >> + host invokes this command after allocating a buffer that can be > >> + accessed by remote host. The allocated address should be programmed > >> + in the ADDRESS register (64 bit), the size should be programmed in > >> + the SIZE register and the memory window index should be programmed > >> + in the ARGUMENT register. The endpoint on receiving this command > >> + will configure the outbound ATU such that trasaction to MW BAR > >> + will be routed to the address provided by the host. > > > > How do we know which is the MW BAR? I assume "MW" refers to "Memory > > Window". > > right memory window. That's again explained in the "Modeling Constructs" > section below. > > > >> + > >> + CMD_LINK_UP (0x3): Command to indicate an NTB application is > >> + bound to the EP device on the host side. Once the endpoint > >> + receives this command from both the hosts, the endpoint will > >> + raise an LINK_UP event to both the hosts to indicate the hosts > >> + can start communicating with each other. > > > > s/raise an/raise a/ > > > > I guess this "LINK_UP event" is something other than the PCIe DL_Up > > state, because each host has already been communicating with the > > endpoint. Right? Is this LINK_UP a software construct? > > Yeah. This is when an NTB client application is bound to the NTB device. > This is used for handshake between the applications running on the two > hosts. > > Thanks > Kishon
