On 1/4/20 7:48 pm, Dan Williams wrote:
On Sun, Mar 29, 2020 at 10:53 PM Alastair D'Silva <alastair@xxxxxxxxxxx> wrote:
OpenCAPI LPC memory is allocated per link, but each link supports
multiple AFUs, and each AFU can have LPC memory assigned to it.
Is there an OpenCAPI primer to decode these objects and their
associations that I can reference?
There isn't presently a primer that I think addresses these questions
nicely (to my knowledge - Fred might have something he can link to?) -
there are the specs published by the OpenCAPI Consortium at
https://opencapi.org but they're really for hardware implementers.
We should probably expand what's currently documented in
Documentation/userspace-api/accelerators/ocxl.rst generally, and this
series should probably update that to include details on LPC.
To explain the specific objects here:
- A "link" is a point-to-point link between the host CPU, and a single
OpenCAPI card. (We don't currently support cards making use of multiple
links for increased bandwidth, though that is supported from a hardware
point of view.)
- On POWER9, each link appears as a separate PCI domain, with a single
bus, and the card appears as a single device.
- A device can have up to 8 functions, per PCI.
- An Attached Functional Unit (AFU) is the abstraction for a particular
application function. Each PCI function defines the number of AFUs it
has through a set of OpenCAPI-specific DVSECs, max 64 per function. The
ocxl driver handles AFU discovery.
- On the host side, LPC memory is mapped by setting a single BAR for the
whole link, but on the device side, LPC memory is requested on a per-AFU
basis, through an AFU descriptor that is exposed through the
aforementioned DVSECs. Hence the need to loop through the AFUs and get
the total required LPC memory to work out the correct BAR value.
--
Andrew Donnellan OzLabs, ADL Canberra
ajd@xxxxxxxxxxxxx IBM Australia Limited
