On Fri, 2020-03-20 at 23:24 +1100, Daniel Axtens wrote: > Hi Haren, > > This is good documentation. > > > Power9 introduced Virtual Accelerator Switchboard (VAS) which allows > > userspace to communicate with Nest Accelerator (NX) directly. But > > kernel has to establish channel to NX for userspace. This document > > describes user space API that application can use to establish > > communication channel. > > > > Signed-off-by: Sukadev Bhattiprolu <sukadev@xxxxxxxxxxxxx> > > Signed-off-by: Haren Myneni <haren@xxxxxxxxxxxxx> > > --- > > Documentation/powerpc/index.rst | 1 + > > Documentation/powerpc/vas-api.rst | 246 ++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 247 insertions(+) > > create mode 100644 Documentation/powerpc/vas-api.rst > > > > diff --git a/Documentation/powerpc/index.rst b/Documentation/powerpc/index.rst > > index 0d45f0f..afe2d5e 100644 > > --- a/Documentation/powerpc/index.rst > > +++ b/Documentation/powerpc/index.rst > > @@ -30,6 +30,7 @@ powerpc > > syscall64-abi > > transactional_memory > > ultravisor > > + vas-api > > > > .. only:: subproject and html > > > > diff --git a/Documentation/powerpc/vas-api.rst b/Documentation/powerpc/vas-api.rst > > new file mode 100644 > > index 0000000..13ce4e7 > > --- /dev/null > > +++ b/Documentation/powerpc/vas-api.rst > > @@ -0,0 +1,246 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > +.. _VAS-API: > > + > > +=================================================== > > +Virtual Accelerator Switchboard (VAS) userspace API > > +=================================================== > > + > > +Introduction > > +============ > > + > > +Power9 processor introduced Virtual Accelerator Switchboard (VAS) which > > +allows both userspace and kernel communicate to co-processor > > +(hardware accelerator) referred to as the Nest Accelerator (NX). The NX > > +unit comprises of one or more hardware engines or co-processor types > > +such as 842 compression, GZIP compression and encryption. On power9, > > +userspace applications will have access to only GZIP Compression engine > > +which supports ZLIB and GZIP compression algorithms in the hardware. > > + > > +To communicate with NX, kernel has to establish a channel or window and > > +then requests can be submitted directly without kernel involvement. > > +Requests to the GZIP engine must be formatted as a co-processor Request > > +Block (CRB) and these CRBs must be submitted to the NX using COPY/PASTE > > +instructions to paste the CRB to hardware address that is associated with > > +the engine's request queue. > > + > > +The GZIP engine provides two priority levels of requests: Normal and > > +High. Only Normal requests are supported from userspace right now. > > + > > +This document explains userspace API that is used to interact with > > +kernel to setup channel / window which can be used to send compression > > +requests directly to NX accelerator. > > + > > + > > +Overview > > +======== > > + > > +Application access to the GZIP engine is provided through > > +/dev/crypto/nx-gzip device node implemented by the VAS/NX device driver. > > +An application must open the /dev/crypto/nx-gzip device to obtain a file > > +descriptor (fd). Then should issue VAS_TX_WIN_OPEN ioctl with this fd to > > +establish connection to the engine. It means send window is opened on GZIP > > +engine for this process. Once a connection is established, the application > > +should use the mmap() system call to map the hardware address of engine's > > +request queue into the application's virtual address space. > > + > > +The application can then submit one or more requests to the the engine by > > +using copy/paste instructions and pasting the CRBs to the virtual address > > +(aka paste_address) returned by mmap(). User space can close the > > +established connection or send window by closing the file descriptior > > +(close(fd)) or upon the process exit. > > + > > +Note that applications can send several requests with the same window or > > +can establish multiple windows, but one window for each file descriptor. > > + > > +Following sections provide additional details and references about the > > +individual steps. > > + > > +NX-GZIP Device Node > > +=================== > > + > > +There is one /dev/crypto/nx-gzip node in the system and it provides > > +access to all GZIP engines in the system. The only valid operations on > > +/dev/crypto/nx-gzip are: > > + > > + * open() the device for read and write. > > + * issue VAS_TX_WIN_OPEN ioctl > > + * mmap() the engine's request queue into application's virtual > > + address space (i.e. get a paste_address for the co-processor > > + engine). > > + * close the device node. > > + > > +Other file operations on this device node are undefined. > > + > > +Note that the copy and paste operations go directly to the hardware and > > +do not go through this device. Refer COPY/PASTE document for more > > +details. > > + > > +Although a system may have several instances of the NX co-processor > > +engines (typically, one per P9 chip) there is just one > > +/dev/crypto/nx-gzip device node in the system. When the nx-gzip device > > +node is opened, Kernel opens send window on a suitable instance of NX > > +accelerator. It finds CPU on which the user process is executing and > > +determine the NX instance for the corresponding chip on which this CPU > > +belongs. > > + > > +Applications may chose a specific instance of the NX co-processor using > > +the vas_id field in the VAS_TX_WIN_OPEN ioctl as detailed below. > > + > > +A userspace library libnxz is available here but still in development: > > + https://github.com/abalib/power-gzip > > + > > +Applications that use inflate / deflate calls can link with libnxz > > +instead of libz and use NX GZIP compression without any modification. > > + > > +Open /dev/crypto/nx-gzip > > +======================== > > + > > +The nx-gzip device should be opened for read and write. No special > > +privileges are needed to open the device. Each window coreesponds to one > > s/coreesponds/corresponds/ > > > +file descriptor. So if the userspace process needs multiple windows, > > +several open calls have to be issued. > > + > > +See open(2) system call man pages for other details such as return values, > > +error codes and restrictions. > > +codes and restrictions. > > You have 'codes and restrictions' twice here. > > > + > > +VAS_TX_WIN_OPEN ioctl > > +===================== > > + > > +Applications should use the VAS_TX_WIN_OPEN ioctl as follows to establish > > +a connection with NX co-processor engine: > > + > > + :: > > + struct vas_tx_win_open_attr { > > + __u32 version; > > + __s16 vas_id; /* specific instance of vas or -1 > > + for default */ > > + __u16 reserved1; > > + __u64 flags; /* For future use */ > > + __u64 reserved2[6]; > > + }; > > + > > + version: The version field must be currently set to 1. > > + vas_id: If '-1' is passed, kernel will make a best-effort attempt > > + to assign an optimal instance of NX for the process. To > > + select the specific VAS instance, refer > > + "Discovery of available VAS engines" section below. > > + > > + flags, reserved1 and reserved2[6] fields are for future extension > > + and must be set to 0. > > + > > + The attributes attr for the VAS_TX_WIN_OPEN ioctl are defined as > > + follows: > > + #define VAS_MAGIC 'v' > > + #define VAS_TX_WIN_OPEN _IOW(VAS_MAGIC, 1, > > + struct vas_tx_win_open_attr) > > + > > + struct vas_tx_win_open_attr attr; > > + rc = ioctl(fd, VAS_TX_WIN_OPEN, &attr); > > + > > + The VAS_TX_WIN_OPEN ioctl returns 0 on success. On errors, it > > + returns -1 and sets the errno variable to indicate the error. > > + > > + Error conditions: > > + EINVAL fd does not refer to a valid VAS device. > > + EINVAL Invalid vas ID > > + EINVAL version is not set with proper value > > + EEXIST Window is already opened for the given fd > > + ENOMEM Memory is not available to allocate window > > + ENOSPC System has too many active windows (connections) > > + opened > > + EINVAL reserved fields are not set to 0. > > + > > + See the ioctl(2) man page for more details, error codes and > > + restrictions. > > + > > +mmap() NX-GZIP device > > +===================== > > + > > +The mmap() system call for a NX-GZIP device fd returns a paste_address > > +that the application can use to copy/paste its CRB to the hardware engines. > > + :: > > + > > + paste_addr = mmap(addr, size, prot, flags, fd, offset); > > + > > + Only restrictions on mmap for a NX-GZIP device fd are: > > + * size should be 4K page size > > Patch 3 seems to allow a 64k page if the system is compiled for 64k > pages... Should it restrict it to 4K? Yes restricted to 4K for NX-GZIP
