On 7/4/2019 2:28 AM, Vladimir Melnik wrote:
So, the disk is OK and the network is OK, I'm 100% sure.
Seems to be a GlusterFS-related issue. Either something needs to be
tweaked or it's a normal performance for a replica-3 cluster.
There is more to it than Gluster on that particular test.
I have some addititional datapoints, since those numbers seemed low
given the long time I have played with Gluster (first install was 3.3)
So I ran that exact test on some locally mounted hard drive sets (mdadm
RAID1- spinning metal) on Cent7 (stock) and U18(stock) and got the
following:
No Gluster involved.
# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s
10+0 records in
That was reproducible over several machines with different CPUs that we
have in production.
Performance is about 20% better when 7200 rpm drives were involved or
when no RAID was involved but never above 18 MB/s
Performance is also MUCH better when I use oflag=direct (roughly 2x)
However, on a U18 VM Host testbed machine that has a seperate SSD swap
disk I get the following, even though I am writing the test.tmp file to
the metal.
# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s
So something else is going on with that particular test. Clearly,
buffers, elevators, cache etc count despite the oflag setting.
For the record on the Gluster Fuse Mount (2x+1arb Volume) on that VM
Host I do get reduced performance
part of that is due to the gluster network being 2x1G using teaming on
that testbed, so there is a network bottleneck.
# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s
So the gluster fuse mount negates the advantage of that SSD swap disk
along with the obvious network bottleneck.
But clearly we have to all agree on same valid test.
-wk
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
