On Fri, 2015-01-09 at 15:04 -0800, Greg Kroah-Hartman wrote:
> On Wed, Dec 10, 2014 at 11:47:41AM -0800, Sudeep Dutt wrote:
> > This patch introduces the SCIF documentation in the header file
> > and describes the IOCTL interface for user mode. mic_overview.txt
> > is updated with documentation on SCIF and a new document
> > describing SCIF in more details is available in scif_overview.txt.
> >
> > Reviewed-by: Nikhil Rao <nikhil.rao@xxxxxxxxx>
> > Reviewed-by: Ashutosh Dixit <ashutosh.dixit@xxxxxxxxx>
> > Signed-off-by: Sudeep Dutt <sudeep.dutt@xxxxxxxxx>
> > ---
> > Documentation/mic/mic_overview.txt | 28 +-
> > Documentation/mic/scif_overview.txt | 62 ++
> > include/uapi/linux/Kbuild | 1 +
> > include/linux/scif.h | 1132 +++++++++++++++++++++++++++++++++++
> > include/uapi/linux/scif_ioctl.h | 233 +++++++
> > 5 files changed, 1444 insertions(+), 12 deletions(-)
> > create mode 100644 Documentation/mic/scif_overview.txt
> > create mode 100644 include/linux/scif.h
> > create mode 100644 include/uapi/linux/scif_ioctl.h
> >
> > diff --git a/Documentation/mic/mic_overview.txt b/Documentation/mic/mic_overview.txt
> > index 77c5418..1a2f2c8 100644
> > --- a/Documentation/mic/mic_overview.txt
> > +++ b/Documentation/mic/mic_overview.txt
> > @@ -24,6 +24,10 @@ a virtual bus called mic bus is created and virtual dma devices are
> > created on it by the host/card drivers. On host the channels are private
> > and used only by the host driver to transfer data for the virtio devices.
> >
> > +The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
> > +low level communications API across PCIe currently implemented for MIC.
> > +More details are available at scif_overview.txt.
> > +
> > Here is a block diagram of the various components described above. The
> > virtio backends are situated on the host rather than the card given better
> > single threaded performance for the host compared to MIC, the ability of
> > @@ -47,18 +51,18 @@ the fact that the virtio block storage backend can only be on the host.
> > | | | Virtio over PCIe IOCTLs |
> > | | +--------------------------+
> > +-----------+ | | | +-----------+
> > -| MIC DMA | | | | | MIC DMA |
> > -| Driver | | | | | Driver |
> > -+-----------+ | | | +-----------+
> > - | | | | |
> > -+---------------+ | | | +----------------+
> > -|MIC virtual Bus| | | | |MIC virtual Bus |
> > -+---------------+ | | | +----------------+
> > - | | | | |
> > - | +--------------+ | +---------------+ |
> > - | |Intel MIC | | |Intel MIC | |
> > - +---|Card Driver | | |Host Driver | |
> > - +--------------+ | +---------------+-----+
> > +| MIC DMA | | +----------+ | +-----------+ | | MIC DMA |
> > +| Driver | | | SCIF | | | SCIF | | | Driver |
> > ++-----------+ | +----------+ | +-----------+ | +-----------+
> > + | | | | | | |
> > ++---------------+ | +-----+-----+ | +-----+-----+ | +---------------+
> > +|MIC virtual Bus| | |SCIF HW Bus| | |SCIF HW BUS| | |MIC virtual Bus|
> > ++---------------+ | +-----------+ | +-----+-----+ | +---------------+
> > + | | | | | | |
> > + | +--------------+ | | | +---------------+ |
> > + | |Intel MIC | | | | |Intel MIC | |
> > + +---|Card Driver +----+ | | |Host Driver | |
> > + +--------------+ | +----+---------------+-----+
> > | | |
> > +-------------------------------------------------------------+
> > | |
> > diff --git a/Documentation/mic/scif_overview.txt b/Documentation/mic/scif_overview.txt
> > new file mode 100644
> > index 0000000..75549c4
> > --- /dev/null
> > +++ b/Documentation/mic/scif_overview.txt
> > @@ -0,0 +1,62 @@
> > +The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low
> > +level communications API across PCIe currently implemented for MIC. Currently
> > +SCIF provides inter-node communication within a single host platform, where a
> > +node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of
> > +communicating over the PCIe bus while providing an API that is symmetric
> > +across all the nodes in the PCIe network. An important design objective for SCIF
> > +is to deliver the maximum possible performance given the communication
> > +abilities of the hardware. SCIF has been used to implement an offload compiler
> > +runtime and OFED support for MPI implementations for MIC coprocessors.
> > +
> > +==== SCIF API Components ====
> > +The SCIF API has the following parts:
> > +1. Connection establishment using a client server model
> > +2. Byte stream messaging intended for short messages
> > +3. Node enumeration to determine online nodes
> > +4. Poll semantics for detection of incoming connections and messages
> > +5. Memory registration to pin down pages
> > +6. Remote memory mapping for low latency CPU accesses via mmap
> > +7. Remote DMA (RDMA) for high bandwidth DMA transfers
> > +8. Fence APIs for RDMA synchronization
> > +
> > +SCIF exposes the notion of a connection which can be used by peer processes on
> > +nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A
> > +process in a SCIF node initiates a SCIF connection to a peer process on a
> > +different node via a SCIF "endpoint". SCIF endpoints support messaging APIs
> > +which are similar to connection oriented socket APIs. Connected SCIF endpoints
> > +can also register local memory which is followed by data transfer using either
> > +DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and
> > +kernel mode clients which are functionally equivalent.
> > +
> > +==== SCIF Performance for MIC ====
> > +DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus
> > +SCIF shows the performance advantages of SCIF for HPC applications and runtimes.
> > +
> > + Comparison of TCP and SCIF based BW
> > +
> > + Throughput (GB/sec)
> > + 8 + PCIe Bandwidth ******
> > + + TCP ######
> > + 7 + ************************************** SCIF %%%%%%
> > + | %%%%%%%%%%%%%%%%%%%
> > + 6 + %%%%
> > + | %%
> > + | %%%
> > + 5 + %%
> > + | %%
> > + 4 + %%
> > + | %%
> > + 3 + %%
> > + | %
> > + 2 + %%
> > + | %%
> > + | %
> > + 1 +
> > + + ######################################
> > + 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-
> > + 1 10 100 1000 10000 100000
> > + Transfer Size (KBytes)
> > +
> > +SCIF allows memory sharing via mmap(..) between processes on different PCIe
> > +nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap
> > +latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.
> > diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild
> > index 4c94f31..9083b60 100644
> > --- a/include/uapi/linux/Kbuild
> > +++ b/include/uapi/linux/Kbuild
> > @@ -345,6 +345,7 @@ header-y += rtc.h
> > header-y += rtnetlink.h
> > header-y += scc.h
> > header-y += sched.h
> > +header-y += scif_ioctl.h
> > header-y += screen_info.h
> > header-y += sctp.h
> > header-y += sdla.h
> > diff --git a/include/linux/scif.h b/include/linux/scif.h
> > new file mode 100644
> > index 0000000..a0652a6
> > --- /dev/null
> > +++ b/include/linux/scif.h
> > @@ -0,0 +1,1132 @@
> > +/*
> > + * Intel MIC Platform Software Stack (MPSS)
> > + *
> > + * This file is provided under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license.
> > + *
> > + * GPL LICENSE SUMMARY
> > + *
> > + * Copyright(c) 2014 Intel Corporation.
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of version 2 of the GNU General Public License as
> > + * published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful, but
> > + * WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> > + * General Public License for more details.
> > + *
> > + * BSD LICENSE
> > + *
> > + * Copyright(c) 2014 Intel Corporation.
> > + *
> > + * Redistribution and use in source and binary forms, with or without
> > + * modification, are permitted provided that the following conditions
> > + * are met:
> > + *
> > + * * Redistributions of source code must retain the above copyright
> > + * notice, this list of conditions and the following disclaimer.
> > + * * Redistributions in binary form must reproduce the above copyright
> > + * notice, this list of conditions and the following disclaimer in
> > + * the documentation and/or other materials provided with the
> > + * distribution.
> > + * * Neither the name of Intel Corporation nor the names of its
> > + * contributors may be used to endorse or promote products derived
> > + * from this software without specific prior written permission.
> > + *
> > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> > + *
> > + * Intel SCIF driver.
> > + *
> > + */
> > +#ifndef __SCIF_H__
> > +#define __SCIF_H__
> > +
> > +#include <linux/types.h>
> > +#include <linux/poll.h>
> > +#include <linux/scif_ioctl.h>
> > +
> > +#define SCIF_ACCEPT_SYNC 1
> > +#define SCIF_SEND_BLOCK 1
> > +#define SCIF_RECV_BLOCK 1
> > +
> > +enum {
> > + SCIF_PROT_READ = (1 << 0),
> > + SCIF_PROT_WRITE = (1 << 1)
> > +};
> > +
> > +enum {
> > + SCIF_MAP_FIXED = 0x10,
> > + SCIF_MAP_KERNEL = 0x20,
> > +};
> > +
> > +enum {
> > + SCIF_FENCE_INIT_SELF = (1 << 0),
> > + SCIF_FENCE_INIT_PEER = (1 << 1),
> > + SCIF_SIGNAL_LOCAL = (1 << 4),
> > + SCIF_SIGNAL_REMOTE = (1 << 5)
> > +};
> > +
> > +enum {
> > + SCIF_RMA_USECPU = (1 << 0),
> > + SCIF_RMA_USECACHE = (1 << 1),
> > + SCIF_RMA_SYNC = (1 << 2),
> > + SCIF_RMA_ORDERED = (1 << 3)
> > +};
> > +
> > +/* End of SCIF Admin Reserved Ports */
> > +#define SCIF_ADMIN_PORT_END 1024
> > +
> > +/* End of SCIF Reserved Ports */
> > +#define SCIF_PORT_RSVD 1088
> > +
> > +typedef struct scif_endpt *scif_epd_t;
> > +
> > +#define SCIF_OPEN_FAILED ((scif_epd_t)-1)
> > +#define SCIF_REGISTER_FAILED ((off_t)-1)
> > +#define SCIF_MMAP_FAILED ((void *)-1)
> > +
> > +/**
> > + * scif_open - Create an endpoint
> > + *
> > + *\return
> > + * Upon successful completion, scif_open() returns an endpoint descriptor to
> > + * be used in subsequent SCIF functions calls to refer to that endpoint;
> > + * otherwise: in user mode SCIF_OPEN_FAILED (that is ((scif_epd_t)-1)) is
> > + * returned and errno is set to indicate the error; in kernel mode a NULL
> > + * scif_epd_t is returned.
> > + *
> > + *\par Errors:
> > + *- ENOMEM
> > + * - Insufficient kernel memory was available
> > + */
>
> Documentation is great, but if you are going to do it, use the proper
> kerneldoc format and not some other odd variant that I have never seen
> before. This whole patch has tons of oddly labled comments, like this:
>
This file was set up for being an input into doxygen some time back. I
agree that it looks really odd so will fix it up and send across a v2
early next week.
Thanks,
Sudeep Dutt
--
To unsubscribe from this list: send the line "unsubscribe linux-doc" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
