On 01/25/2016 11:19 PM, Chuck Lever wrote: > I'd like to propose a discussion of how to take advantage of > persistent memory in network-attached storage scenarios. > > RDMA runs on high speed network fabrics and offloads data > transfer from host CPUs. Thus it is a good match to the > performance characteristics of persistent memory. > > Today Linux supports iSER, SRP, and NFS/RDMA on RDMA > fabrics. What kind of changes are needed in the Linux I/O > stack (in particular, storage targets) and in these storage > protocols to get the most benefit from ultra-low latency > storage? > > There have been recent proposals about how storage protocols > and implementations might need to change (eg. Tom Talpey's > SNIA proposals for changing to a push data transfer model, > Sagi's proposal to utilize DAX under the NFS/RDMA server, > and my proposal for a new pNFS layout to drive RDMA data > transfer directly). > > The outcome of the discussion would be to understand what > people are working on now and what is the desired > architectural approach in order to determine where storage > developers should be focused. > > This could be either a BoF or a session during the main > tracks. There is sure to be a narrow segment of each > track's attendees that would have interest in this topic. > I would like to attend this talk, and also talk about a target we have been developing / utilizing that we would like to propose as a Linux standard driver. (It would be very important for me to also attend the other pmem talks in LSF, as well as some of the MM and FS talks proposed so far) RDMA passive target ~~~~~~~~~~~~~~~~~~~ The idea is to have a storage brick that exports a very low level pure RDMA API to access its memory based storage. The brick might be battery backed volatile based memory, or pmem based. In any case the brick might utilize a much higher capacity then memory by utilizing a "tiering" to slower media, which is enabled by the API. The API is simple: 1. Alloc_2M_block_at_virtual_address (ADDR_64_BIT) ADDR_64_BIT is any virtual address and defines the logical ID of the block. If the ID is already allocated an error is returned. If storage is exhausted return => ENOSPC 2. Free_2M_block_at_virtual_address (ADDR_64_BIT) Space for logical ID is returned to free store and the ID becomes free for a new allocation. 3. map_virtual_address(ADDR_64_BIT, flags) => RDMA handle previously allocated virtual address is locked in memory and an RDMA handle is returned. Flags: read-only, read-write, shared and so on... 4. unmap__virtual_address(ADDR_64_BIT) At this point the brick can write data to slower storage if memory space is needed. The RDMA handle from [3] is revoked. 5. List_mapped_IDs An extent based list of all allocated ranges. (This is usually used on mount or after a crash) The dumb brick is not the Network allocator / storage manager at all. and it is not a smart target / server. like an iser-target or pnfs-DS. A SW defined application can do that, on top of the Dumb-brick. The motivation is a low level very low latency API+library, which can be built upon for higher protocols or used directly for very low latency cluster. It does however mange a virtual allocation map of logical to physical mapping of the 2M blocks. Currently both drivers initiator and target are in Kernel, but with latest advancement by Dan Williams it can be implemented in user-mode as well, Almost. The almost is because: 1. If the target is over a /dev/pmemX then all is fine we have 2M contiguous memory blocks. 2. If the target is over an FS, we have a proposal pending for an falloc_2M_flag to ask the FS for a contiguous 2M allocations only. If any of the 2M allocations fail then return ENOSPC from falloc. This way we guaranty that each 2M block can be mapped by a single RDAM handle. An FS for this purpose is nice for an over-allocated / dynamic space usage by a target and other resources in the server. RDMA Initiator ~~~~~~~~~~~~~~~~~~~ The initiator is just a simple library. Both usermode and Kernel side should be available, for direct access to the RDMA-passive-brick. Thanks. Boaz > -- > Chuck Lever > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
