On Wed, 2013-08-21 at 10:51 -0700, Sudeep Dutt wrote: > ChangeLog: > ========= > > v2 => v3: > a) Patch 1 data structure cleanups, header file include cleanups, > IDA interface reuse and switching to device_create_with_groups(..) > as per feedback from Greg Kroah-Hartman. > b) Patch 7 signal documentation, sleep workaround removal and sysfs > access API cleanups as per feedback from Michael S. Tsirkin. > > v1 => v2: @ http://lwn.net/Articles/563131/ > a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring > into 3 smaller patches and function renames, as per feedback from > Greg Kroah-Hartman. > b) Use VRINGH infrastructure for accessing virtio rings from the host > in patch 5, as per feedback from Michael S. Tsirkin. > > v1: Initial post @ https://lwn.net/Articles/561314/ > Hi Greg, It has been a couple of weeks since we posted v3 @ https://lwn.net/Articles/564795 which addresses all feedback received so far. Have you had a chance to review this version? Please let us know your thoughts. Thanks in advance, Sudeep Dutt > Description: > ============ > > An Intel MIC X100 device is a PCIe form factor add-in coprocessor > card based on the Intel Many Integrated Core (MIC) architecture > that runs a Linux OS. It is a PCIe endpoint in a platform and therefore > implements the three required standard address spaces i.e. configuration, > memory and I/O. The host OS loads a device driver as is typical for > PCIe devices. The card itself runs a bootstrap after reset that > transfers control to the card OS downloaded from the host driver. > The card OS as shipped by Intel is a Linux kernel with modifications > for the X100 devices. > > Since it is a PCIe card, it does not have the ability to host hardware > devices for networking, storage and console. We provide these devices > on X100 coprocessors thus enabling a self-bootable equivalent environment > for applications. A key benefit of our solution is that it leverages > the standard virtio framework for network, disk and console devices, > though in our case the virtio framework is used across a PCIe bus. > > Here is a block diagram of the various components described above. The > virtio backends are situated on the host rather than the card given better > single threaded performance for the host compared to MIC, the ability of > the host to initiate DMA's to/from the card using the MIC DMA engine and > the fact that the virtio block storage backend can only be on the host. > > +----------+ | +----------+ > | Card OS | | | Host OS | > +----------+ | +----------+ > | > +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ > | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio | > | Net | |Console | |Block | | |Net | |Console | |Block | > | Driver| |Driver | |Driver| | |backend | |backend | |backend | > +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ > | | | | | | | > | | | |User | | | > | | | |------|------------|---------|------- > +-------------------+ |Kernel +--------------------------+ > | | | Virtio over PCIe IOCTLs | > | | +--------------------------+ > +--------------+ | | > |Intel MIC | | +---------------+ > |Card Driver | | |Intel MIC | > +--------------+ | |Host Driver | > | | +---------------+ > | | | > +-------------------------------------------------------------+ > | | > | PCIe Bus | > +-------------------------------------------------------------+ > > The following series of patches are partitioned as follows: > > Patch 1: This patch introduces the "Intel MIC Host Driver" in the block > diagram which does the following: > a) Initializes the Intel MIC X100 PCIe devices. > b) Provides sysfs entries for family and stepping information. > > Patch 2: This patch enables the following features in the > "Intel MIC Host Driver" in the block diagram: > a) MSIx, MSI and legacy interrupt support. > b) System Memory Page Table(SMPT) support. SMPT enables system memory > access from the card. On X100 devices the host can program 32 SMPT > registers each capable of accessing 16GB of system memory > address space from X100 devices. The registers can thereby be used > to access a cumulative 512GB of system memory address space from > X100 devices at any point in time. > > Patch 3: This patch enables the following features in the > "Intel MIC Host Driver" in the block diagram: > a) Boots and shuts down the card via sysfs entries. > b) Allocates and maps a device page for communication with the > card driver and updates the device page address via scratchpad > registers. > c) Provides sysfs entries for shutdown status, kernel command line, > firmware, ramdisk, bootmode and log buffer information. > > Patch 4: This patch introduces the "Intel MIC Card Driver" in the block > diagram which does the following: > a) Initializes the Intel MIC X100 platform device and driver. > b) Sets up support to handle shutdown requests from the host. > c) Maps the device page after obtaining the device page address > from the scratchpad registers updated by the host. > d) Informs the host upon a card crash by registering a panic notifier. > e) Informs the host upon a poweroff/halt event. > > Patch 5: This patch introduces the host "Virtio over PCIe" interface for > Intel MIC. It allows creating user space backends on the host and instantiating > virtio devices for them on the Intel MIC card. It uses the existing VRINGH > infrastructure in the kernel to access virtio rings from the host. A character > device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the > user space backend to: > (a) add/remove a virtio device via a device page. > (b) map (R/O) virtio rings and device page to user space. > (c) poll for availability of data. > (d) copy a descriptor or entire descriptor chain to/from the card. > (e) modify virtio configuration. > (f) handle virtio device reset. > The buffers are copied over using CPU copies for this initial patch > and host initiated MIC DMA support is planned for future patches. > The avail and desc virtio rings are in host memory and the used ring > is in card memory to maximize writes across PCIe for performance. > > Patch 6: This patch introduces the card "Virtio over PCIe" interface for > Intel MIC. It allows virtio drivers on the card to communicate with their > user space backends on the host via a device page. Ring 3 apps on the host > can add, remove and configure virtio devices. A thin MIC specific > virtio_config_ops is implemented which is borrowed heavily from previous > similar implementations in lguest and s390 @ > drivers/lguest/lguest_device.c > drivers/s390/kvm/kvm_virtio.c > > Patch 7: This patch introduces a sample user space daemon which > implements the virtio device backends on the host. The daemon > creates/removes/configures virtio device backends by communicating with > the Intel MIC Host Driver. The virtio devices currently supported are > virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. > The daemon also monitors card shutdown status and takes appropriate actions > like killing the virtio backends and resetting the card upon card shutdown > and crashes. > > The patches have been compiled/validated against v3.11-rc5. > > Ashutosh Dixit (2): > Intel MIC Host Driver Changes for Virtio Devices. > Intel MIC Card Driver Changes for Virtio Devices. > > Caz Yokoyama (1): > Sample Implementation of Intel MIC User Space Daemon. > > Dasaratharaman Chandramouli (1): > Intel MIC Host Driver Interrupt/SMPT support. > > Sudeep Dutt (3): > Intel MIC Host Driver for X100 family. > Intel MIC Host Driver, card OS state management. > Intel MIC Card Driver for X100 family. > > Documentation/ABI/testing/sysfs-class-mic.txt | 147 +++ > Documentation/mic/mic_overview.txt | 49 + > Documentation/mic/mpssd/.gitignore | 1 + > Documentation/mic/mpssd/Makefile | 19 + > Documentation/mic/mpssd/micctrl | 173 +++ > Documentation/mic/mpssd/mpss | 202 +++ > Documentation/mic/mpssd/mpssd.c | 1701 +++++++++++++++++++++++++ > Documentation/mic/mpssd/mpssd.h | 100 ++ > Documentation/mic/mpssd/sysfs.c | 102 ++ > drivers/misc/Kconfig | 1 + > drivers/misc/Makefile | 1 + > drivers/misc/mic/Kconfig | 39 + > drivers/misc/mic/Makefile | 6 + > drivers/misc/mic/card/Makefile | 11 + > drivers/misc/mic/card/mic_debugfs.c | 130 ++ > drivers/misc/mic/card/mic_device.c | 306 +++++ > drivers/misc/mic/card/mic_device.h | 133 ++ > drivers/misc/mic/card/mic_virtio.c | 631 +++++++++ > drivers/misc/mic/card/mic_virtio.h | 77 ++ > drivers/misc/mic/card/mic_x100.c | 256 ++++ > drivers/misc/mic/card/mic_x100.h | 48 + > drivers/misc/mic/common/mic_device.h | 51 + > drivers/misc/mic/host/Makefile | 13 + > drivers/misc/mic/host/mic_boot.c | 185 +++ > drivers/misc/mic/host/mic_debugfs.c | 496 +++++++ > drivers/misc/mic/host/mic_device.h | 300 +++++ > drivers/misc/mic/host/mic_fops.c | 221 ++++ > drivers/misc/mic/host/mic_fops.h | 32 + > drivers/misc/mic/host/mic_main.c | 1098 ++++++++++++++++ > drivers/misc/mic/host/mic_smpt.c | 440 +++++++ > drivers/misc/mic/host/mic_smpt.h | 98 ++ > drivers/misc/mic/host/mic_sysfs.c | 468 +++++++ > drivers/misc/mic/host/mic_virtio.c | 704 ++++++++++ > drivers/misc/mic/host/mic_virtio.h | 138 ++ > drivers/misc/mic/host/mic_x100.c | 573 +++++++++ > drivers/misc/mic/host/mic_x100.h | 99 ++ > include/uapi/linux/Kbuild | 2 + > include/uapi/linux/mic_common.h | 238 ++++ > include/uapi/linux/mic_ioctl.h | 74 ++ > 39 files changed, 9363 insertions(+) > create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt > create mode 100644 Documentation/mic/mic_overview.txt > create mode 100644 Documentation/mic/mpssd/.gitignore > create mode 100644 Documentation/mic/mpssd/Makefile > create mode 100755 Documentation/mic/mpssd/micctrl > create mode 100755 Documentation/mic/mpssd/mpss > create mode 100644 Documentation/mic/mpssd/mpssd.c > create mode 100644 Documentation/mic/mpssd/mpssd.h > create mode 100644 Documentation/mic/mpssd/sysfs.c > create mode 100644 drivers/misc/mic/Kconfig > create mode 100644 drivers/misc/mic/Makefile > create mode 100644 drivers/misc/mic/card/Makefile > create mode 100644 drivers/misc/mic/card/mic_debugfs.c > create mode 100644 drivers/misc/mic/card/mic_device.c > create mode 100644 drivers/misc/mic/card/mic_device.h > create mode 100644 drivers/misc/mic/card/mic_virtio.c > create mode 100644 drivers/misc/mic/card/mic_virtio.h > create mode 100644 drivers/misc/mic/card/mic_x100.c > create mode 100644 drivers/misc/mic/card/mic_x100.h > create mode 100644 drivers/misc/mic/common/mic_device.h > create mode 100644 drivers/misc/mic/host/Makefile > create mode 100644 drivers/misc/mic/host/mic_boot.c > create mode 100644 drivers/misc/mic/host/mic_debugfs.c > create mode 100644 drivers/misc/mic/host/mic_device.h > create mode 100644 drivers/misc/mic/host/mic_fops.c > create mode 100644 drivers/misc/mic/host/mic_fops.h > create mode 100644 drivers/misc/mic/host/mic_main.c > create mode 100644 drivers/misc/mic/host/mic_smpt.c > create mode 100644 drivers/misc/mic/host/mic_smpt.h > create mode 100644 drivers/misc/mic/host/mic_sysfs.c > create mode 100644 drivers/misc/mic/host/mic_virtio.c > create mode 100644 drivers/misc/mic/host/mic_virtio.h > create mode 100644 drivers/misc/mic/host/mic_x100.c > create mode 100644 drivers/misc/mic/host/mic_x100.h > create mode 100644 include/uapi/linux/mic_common.h > create mode 100644 include/uapi/linux/mic_ioctl.h > -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
