3.7.8 had write performance problems. Can you try with 3.7.9? On Sat, Mar 26, 2016 at 7:52 PM, Andreas Mather <andreas@xxxxxxxxxxxxxxx> wrote: > Hello! > > I experience very slow performance on gluster 3.7.8 on CentOS 7.2 when using > small block sizes with dd. > > I run 2 gluster servers with 2 bricks each (each located on separate SSDs) > and 2 gluster clients (which will run virtual machines). It's a completely > fresh setup, so no load on the systems. The systems are connected via 1 GBit > network. iperf3 shows 940 MBit throughput, ping shows less than 1ms latency, > so network should be fine. > > The volume is a stripe 2 replica 2 volume. I've only set the volume options > for ping-timeout and owner-uid/gid (as required by qemu). The volume is > mounted via fuse client. > > He're my stats as seen from a client (370MB test file). 'storage1' is one of > the glusterfs servers which also has a fuse mount for these tests: > > > # read (always done echo 3 > /proc/sys/vm/drop_caches before running tests) > > $ dd if=/glustermount/testfile of=/localdisk/testfile bs=4K > 69.7 MB/s > > $ dd if=/glustermount/testfile of=/localdisk/testfile bs=8K > 78.8 MB/s > > $ scp storage1:/glustermount/testfile /localdisk/testfile > 83.3 MB/s > > $ ssh storage1 'dd if=/glustermount/testfile bs=8K' | dd > of=/localdisk/testfile bs=8K > 93.3 MB/s > > > # write > > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=4K > 3.8 MB/s > > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8K > 6.9 MB/s > > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8M > 58.8 MB/s > > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=64M > 58.8 MB/s > > $ scp /localdisk/testfile storage1:/localdisk/testfile > 94 MB/s > > $ scp /localdisk/testfile storage1:/glustermount/testfile > 980 KB/s > > # immediately ran after the previous one. what happened?!?! > $ scp /localdisk/testfile storage1:/glustermount/testfile > 75 MB/s > > $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd of=/localdisk/testfile > bs=8K' > 96.6 MB/s > > $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd > of=/glustermount/testfile bs=8K' > 225 KB/s > > Is this really as good as it gets with gluster? I've tried the following > setups in order to improve write speed for small block sizes: > > - tuning stripe-size. smaller (16KB) and larger (4MB) values > - 'replica 2' only setup (i.e. no striping) > - write-behind on/off > - larger write-behind-window-size > - larger cache-size > - more io-threads > > None of these helped or even changed the numbers much. I've also tried a few > sysctl tweaks, but still not much change. > > So I'd like to know: > > - is this normal speed/behaviour? > - what are the reasons for this? > - can anything be done to improve performance? > - is running VMs on top of gluster a supported/intended use case at all? > > Thanks a lot, > Andreas > > > > > > > > > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
