On Mon, 17 Jun 2013 11:48:18 +0000 "Myklebust, Trond" <Trond.Myklebust@xxxxxxxxxx> wrote: > > On Jun 17, 2013, at 7:01 AM, Jeff Layton <jlayton@xxxxxxxxxx> > wrote: > > > On Sat, 15 Jun 2013 05:09:55 +0000 > > "Myklebust, Trond" <Trond.Myklebust@xxxxxxxxxx> wrote: > > > >>> -----Original Message----- > >>> From: linux-nfs-owner@xxxxxxxxxxxxxxx [mailto:linux-nfs- > >>> owner@xxxxxxxxxxxxxxx] On Behalf Of Jeff Layton > >>> Sent: Friday, June 14, 2013 3:22 PM > >>> To: Sandeep Joshi > >>> Cc: J. Bruce Fields; linux-nfs@xxxxxxxxxxxxxxx > >>> Subject: Re: why does nfsd write not use splice > >>> > >>> On Fri, 14 Jun 2013 17:39:12 +0530 > >>> Sandeep Joshi <sanjos100@xxxxxxxxx> wrote: > >>> > >>>> On Wed, Jun 12, 2013 at 10:16 PM, J. Bruce Fields <bfields@xxxxxxxxxxxx> > >>> wrote: > >>>>> > >>>>> On Wed, Jun 12, 2013 at 09:51:09PM +0530, Sandeep Joshi wrote: > >>>>>> Splice can be implemented independent of RDMA. It is supposed to > >>>>>> transfer pages between two file descriptors. I found some > >>>>>> postings on lkml from > >>>>>> 2006 where Linus says it is quite possible to splice from a socket > >>>>>> to a file. > >>>>>> > >>>>>> See the paragraph: > >>>>>> " For filesystems, splice support tends to be really easy (both > >>>>>> read and write). For other things, it depends a bit. But unlike > >>>>>> sendfile(), it really is quite possible to splice _from_ a socket > >>>>>> too, not just _to_ a socket. But no, that case hasn't been written yet." > >>>>>> http://yarchive.net/comp/linux/splice.html > >>>>>> > >>>>>> Larry McVoy's 1997 proposal for adding splice support to the > >>>>>> kernel can be read at > >>>>>> ftp.tux.org/pub/sites/ftp.bitmover.com/pub/*splice*.*ps*.gz<http:/ > >>>>>> /ftp.tux.org/pub/sites/ftp.bitmover.com/pub/splice.ps.gz> > >>>>>> > >>>>>> Perhaps I should have opened this thread on lkml to determine if > >>>>>> splice from socket to file is still feasible.. > >>>>> > >>>>> Right, the thing is, nfsd reads the rpc request from the socket into > >>>>> its own buffers before it parses it. If you want to move the data > >>>>> directly out of the network buffers into the page cache, then you > >>>>> have to know at what point the write data starts in the > >>>>> request--which I believe will mean doing the xdr parsing (and gss > >>>>> decryption if necessary) as the request comes in off the wire. > >>>>> > >>>>> That sounds like a lot of work and even if you have someone willing > >>>>> to do the work they'd also need to justify that it's worth it. > >>>>> > >>>>> RDMA may have some protocol support that simplifies this, I don't know. > >>>>> > >>>>> --b. > >>>> > >>>> Hi Bruce, > >>>> > >>>>> nfsd reads the rpc request from the socket into its own buffers before it > >>> parses it. > >>>> > >>>> I am not intimate with the gss code but do you think the > >>>> svc_rqst->rq_pages[] can be spliced ? > >>>> > >>> > >>> Probably not in its current form. The problem is one of alignment. You need > >>> to know where the write data actually starts before doing the receive off the > >>> socket, so you can make sure that it ends up in the correct spot in the pages > >>> you're going to splice in. > >>> > >>> There's also the problem of what to do about WRITE requests that contain > >>> data that isn't page aligned or that's shorter than a page... > >> > >> Finally, there is the minor problem that the data that is actually received by the socket may be encrypted, or may need to be checksummed (krb5i) _before_ you can apply it to the file. That is not a particularly good fit for splice(). > >> > > > > Encryption certainly can be a problem, but integrity isn't necessarily > > one. > > > > Basically the idea would be to receive the data off the socket into a > > set of pages and then splice those into the correct spot in the local > > file. In both the privacy and integrity cases, you just have an extra > > step in between. Privacy *may* mean an extra copy too (though some of > > the crypto routines can decrypt data in place), but handling integrity > > shouldn't. > > > > The tricky parts (I think) are determining how to lay out the received > > data into the pages you eventually want to splice into the file before > > you receive that data in, and how to deal with it when the WRITE > > doesn't cover an entire page. > > Once you've copied the data one time, most of the advantage of splice() is gone, since a copy will then exist in processor cache memory and can be duplicated quickly. > Good point. I'm not sure there's much you can do to avoid at least one copy. When the data is received into the skbuff, the networking layer won't have it aligned for an efficient splice. ISTR that some Intel NICs actually have segmentation hardware that purports to understand NFS read and write headers and can split the header and data into different sets of pages. In principle, you could try to use that capability to ensure that the writedata got DMA'ed into a page aligned buffer that you could then splice in. It wouldn't handle all cases so you'd probably still have to copy sometimes, but it might help when you had page-aligned writes that covered an entire page. As you point out though, all of this is deep voodoo, and you'd have to do a bunch of reengineering before you even had an inkling of how worthwhile it is. -- Jeff Layton <jlayton@xxxxxxxxxx> -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
