i continued my testing with 5 clients, all attached over 100Gbit/s
omni-path via IP over IB. when i run the same iozone benchmark across
all 5 clients where gluster is mounted using the glusterfs client, i get
an aggretated write throughput of only about 400GB/s and an aggregated
read throughput of 1.5GB/s. Each node was writing a single 200Gb file in
16MB chunks and the files where distributed across all three bricks on
the server.
the connection was established over Omnipath for sure, as there is no
other link between the nodes and server.
i have no clue what i'm doing wrong here. i can't believe that this is a
normal performance people would expect to see from gluster. i guess
nobody would be using it if it was this slow.
again, when written dreictly to the xfs filesystem on the bricks, i get
over 6GB/s read and write throughput using the same benchmark.
any advise is appreciated
cheers
Pascal
On 04.04.19 12:03, Pascal Suter wrote:
I just noticed i left the most important parameters out :)
here's the write command with filesize and recordsize in it as well :)
./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w
-+S 0 -s 200G -r 16384k
also i ran the benchmark without direct_io which resulted in an even
worse performance.
i also tried to mount the gluster volume via nfs-ganesha which further
reduced throughput down to about 450MB/s
if i run the iozone benchmark with 3 threads writing to all three
bricks directly (from the xfs filesystem) i get throughputs of around
6GB/s .. if I run the same benchmark through gluster mounted locally
using the fuse client and with enough threads so that each brick gets
at least one file written to it, i end up seing throughputs around
1.5GB/s .. that's a 4x decrease in performance. at it actually is the
same if i run the benchmark with less threads and files only get
written to two out of three bricks.
cpu load on the server is around 25% by the way, nicely distributed
across all available cores.
i can't believe that gluster should really be so slow and everybody is
just happily using it. any hints on what i'm doing wrong are very
welcome.
i'm using gluster 6.0 by the way.
regards
Pascal
On 03.04.19 12:28, Pascal Suter wrote:
Hi all
I am currently testing gluster on a single server. I have three
bricks, each a hardware RAID6 volume with thin provisioned LVM that
was aligned to the RAID and then formatted with xfs.
i've created a distributed volume so that entire files get
distributed across my three bricks.
first I ran a iozone benchmark across each brick testing the read and
write perofrmance of a single large file per brick
i then mounted my gluster volume locally and ran another iozone run
with the same parameters writing a single file. the file went to
brick 1 which, when used driectly, would write with 2.3GB/s and read
with 1.5GB/s. however, through gluster i got only 800MB/s read and
750MB/s write throughput
another run with two processes each writing a file, where one file
went to the first brick and the other file to the second brick (which
by itself when directly accessed wrote at 2.8GB/s and read at
2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated
read throughput.
Is this a normal performance i can expect out of a glusterfs or is it
worth tuning in order to really get closer to the actual brick
filesystem performance?
here are the iozone commands i use for writing and reading.. note
that i am using directIO in order to make sure i don't get fooled by
cache :)
./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0
-s $filesize -r $recordsize > iozone-brick${b}-write.txt
./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0
-s $filesize -r $recordsize > iozone-brick${b}-read.txt
cheers
Pascal
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
