On 09/27/2016 08:53 AM, Raghavendra Gowdappa wrote:
----- Original Message -----
From: "Ric Wheeler" <rwheeler@xxxxxxxxxx>
To: "Raghavendra Gowdappa" <rgowdapp@xxxxxxxxxx>, "Saravanakumar Arumugam" <sarumuga@xxxxxxxxxx>
Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>, "Ben Turner" <bturner@xxxxxxxxxx>, "Ben England"
<bengland@xxxxxxxxxx>
Sent: Tuesday, September 27, 2016 10:51:48 AM
Subject: Re: libgfapi zero copy write - application in samba, nfs-ganesha
On 09/27/2016 07:56 AM, Raghavendra Gowdappa wrote:
+Manoj, +Ben turner, +Ben England.
@Perf-team,
Do you think the gains are significant enough, so that smb and nfs-ganesha
team can start thinking about consuming this change?
regards,
Raghavendra
This is a large gain but I think that we might see even larger gains (a lot
depends on how we implement copy offload :)).
Can you elaborate on what you mean "copy offload"? If it is the way we avoid a copy in gfapi (from application buffer), following is the workflow:
<commit>
Work flow of zero copy write operation:
--------------------------------------
1) Application requests a buffer of specific size. A new buffer is
allocated from iobuf pool, and this buffer is passed on to application.
Achieved using "glfs_get_buffer"
2) Application writes into the received buffer, and passes that to
libgfapi, and libgfapi in turn passes the same buffer to underlying
translators. This avoids a memcpy in glfs write
Achieved using "glfs_zero_write"
3) Once the write operation is complete, Application must take the
responsibilty of freeing the buffer.
Achieved using "glfs_free_buffer"
</commit>
Do you've any suggestions/improvements on this? I think Shyam mentioned an alternative approach (for zero-copy readv I think), let me look up at that too.
regards,
Raghavendra
Both NFS and SMB support a copy offload that allows a client to produce a new
copy of a file without bringing data over the wire. Both, if I remember
correctly, do a ranged copy within a file.
The key here is that since the data does not move over the wire from server to
client, we can shift the performance bottleneck to the storage server.
If we have a slow (1GB) link between client and server, we should be able to do
that copy as if it happened just on the server itself. For a single NFS server
(not a clustered, scale out server), that usually means we are as fast as the
local file system copy.
Note that there are also servers that simply "reflink" that file, so we have a
very small amount of time needed on the server to produce that copy. This can
be a huge win for say a copy of a virtual machine guest image.
Gluster and other distributed servers won't benefit as much as a local server
would I suspect because of the need to do things internally over our networks
between storage server nodes.
Hope that makes my thoughts clearer?
Here is a link to a brief overview of the new Linux system call:
https://kernelnewbies.org/Linux_4.5#head-6df3d298d8e0afa8e85e1125cc54d5f13b9a0d8c
Note that block devices or pseudo devices can also implement a copy offload.
Regards,
Ric
Worth looking at how we can make use of it.
thanks!
Ric
----- Original Message -----
From: "Saravanakumar Arumugam" <sarumuga@xxxxxxxxxx>
To: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>
Sent: Monday, September 26, 2016 7:18:26 PM
Subject: libgfapi zero copy write - application in samba,
nfs-ganesha
Hi,
I have carried out "basic" performance measurement with zero copy write
APIs.
Throughput of zero copy write is 57 MB/sec vs default write 43 MB/sec.
( I have modified Ben England's gfapi_perf_test.c for this. Attached the
same
for reference )
We would like to hear how samba/ nfs-ganesha who are libgfapi users can
make
use of this.
Please provide your comments. Refer attached results.
Zero copy in write patch: http://review.gluster.org/#/c/14784/
Thanks,
Saravana
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
