> -----Original Message----- > From: Tillmann Heidsieck [mailto:theidsieck@xxxxxxxxx] > Sent: Tuesday, August 11, 2015 6:01 AM > To: Yoder Stuart-B08248; gregkh@xxxxxxxxxxxxxxxxxxx; Rivera Jose-B46482; katz Itai-RM05202 > Cc: devel@xxxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; agraf@xxxxxxx; arnd@xxxxxxxx > Subject: Re: [PATCH v2] staging: fsl-mc: add DPAA2 overview readme > > Hi Stuart, > > I am by no means a native speaker, but I have proof-read my fair share of articles and theses, so here are a > bunch of suggestions which might or might not help improve you document. > > On 07.08.2015 03:09, Stuart Yoder wrote: > > add README file providing an overview of the DPAA2 architecture > > and how it is integrated in Linux > > > > Signed-off-by: Stuart Yoder <stuart.yoder@xxxxxxxxxxxxx> > > --- > > -v2: added changelog text > > > > drivers/staging/fsl-mc/README.txt | 364 ++++++++++++++++++++++++++++++++++++++ > > drivers/staging/fsl-mc/TODO | 4 - > > 2 files changed, 364 insertions(+), 4 deletions(-) > > create mode 100644 drivers/staging/fsl-mc/README.txt > > > > diff --git a/drivers/staging/fsl-mc/README.txt b/drivers/staging/fsl-mc/README.txt > > new file mode 100644 > > index 0000000..8214102 > > --- /dev/null > > +++ b/drivers/staging/fsl-mc/README.txt > > @@ -0,0 +1,364 @@ > > +Copyright (C) 2015 Freescale Semiconductor Inc. > > + > > +DPAA2 (Data Path Acceleration Architecture Gen2) > > +------------------------------------------------ > > + > > +This document provides an overview of the Freescale DPAA2 architecture > > +and how it is integrated into the Linux kernel. > > + > > +Contents summary > > + -DPAA2 overview > > + -Overview of DPAA2 objects > > + -DPAA2 Linux driver architecture overview > > + -bus driver > > + -dprc driver > > - DPRC driver > > > + -allocator > > + -dpio driver > > - DPIO driver > > > + -Ethernet > > + -mac > mac -> MAC > > + > > +DPAA2 Overview > > +-------------- > > + > > +DPAA2 is a hardware architecture designed for high-speeed network > > +packet processing. DPAA2 consists of sophisticated mechanisms for > > +processing Ethernet packets, queue management, buffer management, > > +autonomous L2 switching, virtual Ethernet bridging, and accelerator > > +(e.g. crypto) sharing. > > + > > +A DPAA2 hardware component called the Management Complex (or MC) manages the > > +DPAA2 hardware resources. The MC provides an object-based abstraction for > > +software drivers to use the DPAA2 hardware. > > + > > +The MC uses DPAA2 hardware resources such as queues, buffer pools, and > > +network ports to create functional objects/devices such as network > > +interfaces, an L2 switch, or accelerator instances. > > + > > +The MC provides memory-mapped I/O command interfaces (MC portals) > > +which DPAA2 software drivers use to operate on DPAA2 objects: > > + > > + +--------------------------------------+ > > + | OS | > > + | DPAA2 drivers | > > + | | | > > + +-----------------------------|--------+ > > + | > > + | (create,discover,connect > > + | config,use,destroy) > > + | > > + DPAA2 | > > + +------------------------| mc portal |-+ > > + | | | > > + | +- - - - - - - - - - - - -V- - -+ | > > + | | | | > > + | | Management Complex (MC) | | > > + | | | | > > + | +- - - - - - - - - - - - - - - -+ | > > + | | > > + | Hardware Hardware | > > + | Resources Objects | > > + | --------- ------- | > > + | -queues -DPRC | > > + | -buffer pools -DPMCP | > > + | -Eth MACs/ports -DPIO | > > + | -network interface -DPNI | > > + | profiles -DPMAC | > > + | -queue portals -DPBP | > > + | -MC portals ... | > > + | ... | > > + | | > > + +--------------------------------------+ > > + > > +The MC mediates operations such as create, discover, > > +connect, configuration, and destroy. Fast-path operations > > +on data, such as packet transmit/receive, are not mediated by > > +the MC and are done directly using memory mapped regions in > > +DPIO objects. > > + > > +Overview of DPAA2 Objects > > +------------------------- > > +The section provides a brief overview of some key objects > > +in the DPAA2 hardware. A simple scenario is described illustrating > > +the objects involved in creating a network interfaces. > > + > > +-DPRC (Datapath Resource Container) > > + > > + A DPRC is an container object that holds all the other > > ... is a container ... > > > + types of DPAA2 objects. In the example diagram below there > > + are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) > > + in the container. > > + > > + +---------------------------------------------------------+ > > + | DPRC | > > + | | > > + | +-------+ +-------+ +-------+ +-------+ +-------+ | > > + | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | > > + | +-------+ +-------+ +-------+ +---+---+ +---+---+ | > > + | | DPMCP | | DPIO | | > > + | +-------+ +-------+ | > > + | | DPMCP | | > > + | +-------+ | > > + | | > > + +---------------------------------------------------------+ > > + > > + From the point of view of an OS, a DPRC is bus-like. Like > > maybe replace "is bus-like" with "behaves similar to a bus" or "can be viewed as a bus" > > > + a plug-and-play bus, such as PCI, DPRC commands can be used to > > maybe replace "Like a pnp bus..." with "Similar to a plug-and-play bus, such as PCI, DPRC ..." > > > + enumerate the contents of the DPRC, discover the hardware > > + objects present (including mappable regions and interrupts). > > + > > + dprc.1 (bus) > > + | > > + +--+--------+-------+-------+-------+ > > + | | | | | > > + dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 > > + dpmcp.2 dpio.2 > > + dpmcp.3 > > + > > + Hardware objects can be created and destroyed dynamically, providing > > + the ability to hot plug/unplug objects in and out of the DPRC. > > + > > + A DPRC has a mappable mmio region (an MC portal) that can be used > > - mmio -> MMIO (not sure whether mappable MMIO is redundant or not) > - a MC portal > > > + to send MC commands. It has an interrupt for status events (like > > + hotplug). > > + > > + All objects in a container share the same hardware "isolation context". > > + This means that with respect to an IOMMU the isolation granularity > > + is at the DPRC (container) level, not at the individual object > > + level. > > + > > + DPRCs can be defined statically and populated with objects > > + via a config file passed to the MC when firmware starts > > + it. There is also a Linux user space tool called "restool" > > + that can be used to create/destroy containers and objects > > + dynamically. > > + > > +-DPAA2 Objects for an Ethernet Network Interface > > + > > + A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX > > + queuing mechanisms, configuration mechanisms, buffer management, > > + physical ports, and interrupts. DPAA2 uses a more granular approach > > + utilizing multiple hardware objects. Each object has specialized > > + functions, and are used together by software to provide Ethernet network > > Each object provides specialized functions. Groups of these objects are used by the software to provide > Ethernet network interface functionality. > > > + interface functionality. This approach provides efficient use of finite > > + hardware resources, flexibility, and performance advantages. > > + > > + The diagram below shows the objects needed for a simple > > + network interface configuration on a system with 2 CPUs. > > + > > + +---+---+ +---+---+ > > + CPU0 CPU1 > > + +---+---+ +---+---+ > > + | | > > + +---+---+ +---+---+ > > + DPIO DPIO > > + +---+---+ +---+---+ > > + \ / > > + \ / > > + \ / > > + +---+---+ > > + DPNI --- DPBP,DPMCP > > + +---+---+ > > + | > > + | > > + +---+---+ > > + DPMAC > > + +---+---+ > > + | > > + port/PHY > > + > > + Below the objects are described. For each object a brief description > > The objects depicted in this figure are described below. > > > + is provided along with a summary of the kinds of operations the object > > + supports and a summary of key resources of the object (mmio regions > > + and irqs). > > mmio -> MMIO > irqs -> IRQs > > > + > > + -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a > > + hardware device that connects to an Ethernet PHY and allows > > + physical transmission and reception of Ethernet frames. > > + -mmio regions: none > > mmio -> MMIO > > > + -irqs: dpni link change > > irqs -> IRQs > dpni -> DPNI > > > + -commands: set link up/down, link config, get stats, > > + irq config, enable, reset > > irq -> IRQ > > > + > > + -DPNI (Datapath Network Interface): contains TX/RX queues, > > + network interface configuration, and rx buffer pool configuration > > rx -> RX > > > + mechanisms. > > + -mmio regions: none > > mmio -> MMIO > > > + -irqs: link state > > irqs -> IRQs > > > + -commands: port config, offload config, queue config, > > + parse/classify config, irq config, enable, reset > > irq -> IRQ > > > + > > + -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue > > + packets and do hardware buffer pool management operations. For > > + optimum performance there is typically DPIO per CPU. This allows > > For optimum performance there is typically one DPIO assigned to each CPU > > > + each CPU to perform simultaneous enqueue/dequeue operations. > > This allows different CPUs to simultaneously perform enqueue/dequeue operations. > > > + -mmio regions: queue operations, buffer mgmt > > mmio -> MMIO > mgmt -> management > > > + -irqs: data availability, congestion notification, buffer > > + pool depletion > > irqs -> IRQs > > > + -commands: irq config, enable, reset > > irq -> IRQ > > > + > > + -DPBP (Datapath Buffer Pool): represents a hardware buffer > > + pool. > > + -mmio regions: none > > + -irqs: none > > + -commands: enable, reset > > mmio -> MMIO > irqs -> IRQs > > > + > > + -DPMCP (Datapath MC Portal): provides an MC command portal. > > + Used by drivers to send commands to the MC to manage > > + objects. > > + -mmio regions: MC command portal > > + -irqs: command completion > > + -commands: irq config, enable, reset > > mmio -> MMIO > irqs -> IRQs > irq -> IRQ > > > + > > + Object Connections > > + ------------------ > > + Some objects have explicit relationships that must > > + be configured: > > + > > + -DPNI <--> DPMAC > > + -DPNI <--> DPNI > > + -DPNI <--> L2-switch-port > > + A DPNI must be connected to something such as a DPMAC, > > + another DPNI, or L2 switch port. The DPNI connection > > + is made via a DPRC command. > > + > > + +-------+ +-------+ > > + | DPNI | | DPMAC | > > + +---+---+ +---+---+ > > + | | > > + +==========+ > > + > > + -DPNI <--> DPBP > > + A network interface requires a 'buffer pool' (DPBP > > + object) which provides a list of pointers to memory > > + where received Ethernet data is to be copied. The > > + Ethernet driver configures the DPBPs associated with > > + the network interface. > > + > > + Interrupts > > + ---------- > > + All interrupts generated by DPAA2 objects are message > > + interrupts. At the hardware level message interrupts > > + generated by devices will normally have 3 components-- > > + 1) a non-spoofable 'device-id' expressed on the hardware > > + bus, 2) an address, 3) a data value. > > + > > + In the case of DPAA2 devices/objects, all objects in the > > + same container/DPRC share the same 'device-id'. > > + For ARM-based SoC this is the same as the stream ID. > > + > > + > > +DPAA2 Linux Driver Overview > > +--------------------------- > > + > > +This section provides an overview of the Linux kernel drivers for > > +DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" > > +drivers and 2) functional object drivers (such as Ethernet). > > + > > +As described previously, a DPRC is a container that holds the other > > +types of DPAA2 objects. It is functionally similar to a plug-and-play > > +bus controller. > > + > > +Each object in the DPRC is a Linux "device" and is bound to a driver. > > +The diagram below shows the Linux drivers involved in a networking > > +scenario and the objects bound to each driver. A brief description > > +of each driver follows. > > + > > + +------------+ > > + | OS Network | > > + | Stack | > > + +------------+ +------------+ > > + | Allocator |. . . . . . . | Ethernet | > > + |(dpmcp,dpbp)| | (dpni) | > > + +-.----------+ +---+---+----+ > > + . . ^ | > > + . . <data avail, | |<enqueue, > > + . . tx confirm> | | dequeue> > > + +-------------+ . | | > > + | DPRC driver | . +---+---V----+ +---------+ > > + | (dprc) | . . . . . .| DPIO driver| | MAC | > > + +----------+--+ | (dpio) | | (dpmac) | > > + | +------+-----+ +-----+---+ > > + |<dev add/remove> | | > > + | | | > > + +----+--------------+ | +--+---+ > > + | mc-bus driver | | | PHY | > > + | | | |driver| > > + | /fsl-mc@80c000000 | | +--+---+ > > + +-------------------+ | | > > + | | > > + ================================ HARDWARE =========|=================|====== > > + DPIO | > > + | | > > + DPNI---DPBP | > > + | | > > + DPMAC | > > + | | > > + PHY ---------------+ > > + ===================================================|======================== > > + > > +A brief description of each driver is provided below. > > + > > + mc-bus driver > > + ------------- > mc-bus -> MC-Bus or MC-bus > > + The mc-bus driver is a platform driver and is probed from an > > + "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. > > + It is responsible for bootstrapping the DPAA2 kernel infrastructure. > > + Key functions include: > > + -registering a new bus type named "fsl-mc" with the kernel, > > + and implementing bus call-backs (e.g. match/uevent/dev_groups) > > + -implemeting APIs for DPAA2 driver registration and for device > > + add/remove > > + -creates an MSI irq domain > irq -> IRQ > > + -do a device add of the 'root' DPRC device, which is needed > > + to bootstrap things > > + > > + DPRC driver > > + ----------- > > + The dprc-driver is bound DPRC objects and does runtime management > > + of a bus instance. It performs the initial bus scan of the DPRC > > + and handles interrupts for container events such as hot plug. > > + > > + Allocator > > + ---------- > > + Certain objects such as DPMCP and DPBP are generic and fungible, > > + and are intended to be used by other drivers. For example, > > + the DPAA2 Ethernet driver needs: > > + -DPMCPs to send MC commands, to configure network interfaces > > + -DPBPs for network buffer pools > > + > > + The allocator driver registers for these allocatable object types > > + and those objects are bound to the allocator when the bus is probed. > > + The allocator maintains a pool of objects that are available for > > + allocation by other DPAA2 drivers. > > + > > + DPIO driver > > + ----------- > > + The DPIO driver is bound to DPIO objects and provides services that allow > > + other drivers such as the Ethernet driver to receive and transmit data. > > + Key services include: > > + -data availability notifications > > + -hardware queuing operations (enqueue and dequeue of data) > > + -hardware buffer pool management > > + > > + There is typically one DPIO object per physical CPU for optimum > > + performance, allowing each CPU to simultaneously enqueue > > + and dequeue data. > > allowing different CPUs to simultaneously ... > > > + > > + The DPIO driver operates on behalf of all DPAA2 drivers > > + active in the kernel-- Ethernet, crypto, compression, > > + etc. > > + > > + Ethernet > > + -------- > > + The Ethernet driver is bound to a DPNI and implements the kernel > > + interfaces needed to connect the DPAA2 network interface to > > + the network stack. > > + > > + Each DPNI corresponds to a Linux network interface. > > + > > + MAC driver > > + ---------- > > + An Ethernet PHY is an off-chip, board specific component and is managed > > + by the appropriate PHY driver via an mdio bus. The MAC driver > > + plays a role of being a proxy between the PHY driver and the > > + MC. It does this proxy via the MC commands to a DPMAC object. > > diff --git a/drivers/staging/fsl-mc/TODO b/drivers/staging/fsl-mc/TODO > > index c29516b..3894368 100644 > > --- a/drivers/staging/fsl-mc/TODO > > +++ b/drivers/staging/fsl-mc/TODO > > @@ -1,7 +1,3 @@ > > -* Add README file (with ASCII art) describing relationships between > > - DPAA2 objects and how combine them to make a NIC, an LS2 switch, etc. > > - Also, define all acronyms used. > > - > > * Decide if multiple root fsl-mc buses will be supported per Linux instance, > > and if so add support for this. > > Appreciate the detailed review. Like most of your suggestions and will incorporate on the next update to this document. Stuart _______________________________________________ devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxx http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel
