In some basic testing using 2 RDMA gluster servers configured as a replicated pair, I'm seeing better significantly better performance with IPoIB versus RDMA only volumes (almost 2x in write performance and ~10-20% in read). Has anyone had a similar experience? glusterfs 3.4.0 built on Jul 12 2013 20:36:46 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. Volume Name: perftest Type: Replicate Volume ID: 10578349-a466-497a-b7de-1fcd34f967da Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: rdma Bricks: Brick1: cs1-i:/gluster/perftest Brick2: cs2-i:/gluster/perftest cs1-i:/perftest.rdma 3.2T 13G 3.2T 1% /tmp/perftest Run began: Wed Jul 17 08:31:39 2013 Record Size 1024 KB File size set to 5242880 KB Command line used: iozone -i0 -i1 -r 1024#k -s 5#g Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 5242880 1024 422514 496985 364946 324213 Volume Name: perftest Type: Replicate Volume ID: 10578349-a466-497a-b7de-1fcd34f967da Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: cs1-i:/gluster/perftest Brick2: cs2-i:/gluster/perftest cs1-i:/perftest.tcp 3.2T 13G 3.2T 1% /tmp/perftest Run began: Wed Jul 17 08:38:23 2013 Record Size 1024 KB File size set to 5242880 KB Command line used: iozone -i0 -i1 -r 1024#k -s 5#g Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 5242880 1024 557479 549629 667576 666251 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130717/8e0d2b35/attachment.html>
