On Thu, Mar 19, 2015 at 08:00:05AM -0700, Marc Eshel wrote: > linux-nfs-owner@xxxxxxxxxxxxxxx wrote on 03/18/2015 02:11:44 PM: > > > From: "J. Bruce Fields" <bfields@xxxxxxxxxxxx> > > To: Anna Schumaker <Anna.Schumaker@xxxxxxxxxx> > > Cc: linux-nfs@xxxxxxxxxxxxxxx > > Date: 03/18/2015 02:14 PM > > Subject: Re: [PATCH v3 3/3] NFSD: Add support for encoding multiple > segments > > Sent by: linux-nfs-owner@xxxxxxxxxxxxxxx > > > > On Wed, Mar 18, 2015 at 05:03:32PM -0400, Anna Schumaker wrote: > > > On 03/18/2015 04:55 PM, J. Bruce Fields wrote: > > > > On Wed, Mar 18, 2015 at 04:39:24PM -0400, Anna Schumaker wrote: > > > >> On 03/18/2015 02:55 PM, J. Bruce Fields wrote: > > > >>> On Wed, Mar 18, 2015 at 02:16:29PM -0400, Anna Schumaker wrote: > > > >>>> On 03/17/2015 05:36 PM, J. Bruce Fields wrote: > > > >>>>> On Tue, Mar 17, 2015 at 04:07:38PM -0400, J. Bruce Fields wrote: > > > >>>>>> On Tue, Mar 17, 2015 at 03:56:33PM -0400, J. Bruce Fields > wrote: > > > >>>>>>> On Mon, Mar 16, 2015 at 05:18:08PM -0400, Anna Schumaker > wrote: > > > >>>>>>>> This patch implements sending an array of segments back > > to the client. > > > >>>>>>>> Clients should be prepared to handle multiple segment > > reads to make this > > > >>>>>>>> useful. We try to splice the first data segment into the > > XDR result, > > > >>>>>>>> and remaining segments are encoded directly. > > > >>>>>>> > > > >>>>>>> I'm still interested in what would happen if we started with > an > > > >>>>>>> implementation like: > > > >>>>>>> > > > >>>>>>> - if the entire requested range falls within a hole, return > that > > > >>>>>>> single hole. > > > >>>>>>> - otherwise, just treat the thing as one big data segment. > > > >>>>>>> > > > >>>>>>> That would provide a benefit in the case there are large-ish > holes > > > >>>>>>> with minimal impact otherwise. > > > >>>>>>> > > > >>>>>>> (Though patches for full support are still useful even if only > for > > > >>>>>>> client-testing purposes.) > > > >>>>>> > > > >>>>>> Also, looks like > > > >>>>>> > > > >>>>>> xvs_io -c "fiemap -v" <file> > > > >>>>>> > > > >>>>>> will give hole sizes for a given <file>. (Thanks, > > esandeen.) Running > > > >>>>>> that on a few of my test vm images shows a fair number of large > > > >>>>>> (hundreds of megs) files, which suggests identifying only > > >=rwsize holes > > > >>>>>> might still be useful. > > > >>>>> > > > >>>>> Just for fun.... I wrote the following test program and ran it > on my > > > >>>>> collection of testing vm's. Some looked like this: > > > >>>>> > > > >>>>> f21-1.qcow2 > > > >>>>> 144784 -rw-------. 1 qemu qemu 8591507456 Mar 16 10:13 > f21-1.qcow2 > > > >>>>> total hole bytes: 8443252736 (98%) > > > >>>>> in aligned 1MB chunks: 8428453888 (98%) > > > >>>>> > > > >>>>> So, basically, read_plus would save transferring most of thedata > even > > > >>>>> when only handling 1MB holes. > > > >>>>> > > > >>>>> But some looked like this: > > > >>>>> > > > >>>>> 501524 -rw-------. 1 qemu qemu 8589934592 May 20 2014 > > rhel6-1-1.img > > > >>>>> total hole bytes: 8077516800 (94%) > > > >>>>> in aligned 1MB chunks: 0 (0%) > > > >>>>> > > > >>>>> So the READ_PLUS that caught every hole might save a lot, the > one that > > > >>>>> only caught 1MB holes wouldn't help at all. > > > >>>>> > > > >>>>> And there were lots of examples in between those two extremes. > > > >>>> > > > >>>> I tested with three different 512 MB files: 100% data, 100% > > hole, and alternating every megabyte. The results were surprising: > > > >>>> > > > >>>> | v4.1 | v4.2 > > > >>>> ----------------------- > > > >>>> data | 0.685s | 0.714s > > > >>>> hole | 0.485s | 15.547s > > > >>>> mixed | 1.283s | 0.448 > > > >>>> > > > >>>> >From what I can tell, the 100% hole case takes so long because > of the > > > >>>>> SEEK_DATA call in nfsd4_encode_read_plus_hole(). I took this > out to > > > >>>>> trick the function into thinking that the entire file was > already a > > > >>>>> hole, and runtime dropped to the levels of v4.1 and v4.2. > > > >>> > > > >>> Wait, that 15s is due to just one SEEK_DATA? > > > >> > > > >> The server is returning a larger hole than the client can read > > at once, so there are several SEEK_DATA calls made to verify that > > there are no data segments before the end of the file. > > > >> > > > >>> > > > >>>> I wonder > > > >>>>> if this is filesystem dependent? My server is exporting ext4. > > > >>> > > > >>> Sounds like just a bug. I've been doing lots of > lseek(.,.,SEEK_DATA) on > > > >>> both ext4 and xfs without seeing anything that weird. > > > >> > > > >> It looks like something weird on ext4. I switched my exported > > filesystem to xfs: > > > > > > > > Huh. Maybe we should report a bug.... > > > > > > > >> > > > >> | v4.1 | v4.2 > > > >> ------+--------+------- > > > >> data | 0.764s | 1.343s > > > > > > > > That's too bad. Non-sparse files are surely still a common case and > > > > we'd like to not see a slowdown there.... I wonder if we can figure > out > > > > where it's coming from? > > > > > > That's a good question, especially since the 1G file didn't double > > this time. Maybe a VM quirk? > > > > We definitely need to figure it out, I think. If we can't make > > READ_PLUS perform as well as READ (or very close to it) in the > > non-sparse case then I don't think we'll want it, and as Trond suggested > > we may want to consider something more fiemap-like instead. > > Testing Anna's NFS client with the Ganesha NFS server and GPFS file system > shows the same numbers for READ with v4.1 and READ_PUSE with v4.2 of a > data file. Using sparse files READ_PLUS is 5 times faster than READ. Thanks! Is it possible to report the exact numbers? Is Ganesha also implementing READ_PLUS with SEEK_HOLE/SEEK_DATA? If so then maybe the difference is the filesystem. Might be interesting to run the same sort of test with ganesha exporting xfs and/or knfsd exporting GPFS. --b. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
