Poor read performance and other funny performance results

[Flemming Frandsen <dren.dk@xxxxxxxxx>]

Hi, I'm trying to set up a very simple glusterfs based system, but I'm getting some strange performance numbers.

To keep the test simple I've:
* Set up a volume on a single server
* Mounted the volume only via localhost
* Kept the files small (1-10G) so they always fit in page cache.

I hope this means that I'm testing only glusterfs performance and not other aspects of my machine.

The host has two E5-2687W v3 @ 3.10GHz CPUs and 377 GB of ram and nothing else running.

I'm running ubuntu server 16.04 with the glusterfs 3.10 from:
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.10

The script I'm using is:
#!/bin/bash                                                                                                                                                                                        

set -x

# Volume was created using:                                                                                                                                                                        
# gluster volume create testvol transport tcp localhost:/mnt force                                                                                                                                 
# gluster volume start testvol                                                                                                                                                                     



BS=1M
BC=1k


umount hest
mount -t glusterfs localhost:/testvol hest
dd if=/dev/zero of=hest/test/10g bs=$BS count=$BC
dd if=/mnt/test/10g of=/dev/null bs=$BS
dd if=/mnt/test/10g of=/dev/null bs=$BS
dd if=hest/test/10g of=/dev/null bs=$BS
dd if=hest/test/1g of=/dev/null bs=$BS


echo "Purely local test"
dd if=/dev/zero of=/mnt/fest/10g bs=$BS count=$BC
dd if=/mnt/fest/10g of=/dev/null bs=$BS
dd if=/mnt/fest/10g of=/dev/null bs=$BS

The typical results I get are:
+ BS=1M
+ BC=1k
+ umount hest
+ mount -t glusterfs localhost:/testvol hest
+ dd if=/dev/zero of=hest/test/10g bs=1M count=1k
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.824452 s, 1.3 GB/s
+ dd if=/mnt/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.374057 s, 2.9 GB/s
+ dd if=/mnt/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.202726 s, 5.3 GB/s
+ dd if=hest/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.38545 s, 775 MB/s
+ dd if=hest/test/1g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.26429 s, 849 MB/s


Purely local test
+ dd if=/dev/zero of=/mnt/fest/10g bs=1M count=1k
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.957372 s, 1.1 GB/s
+ dd if=/mnt/fest/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.245052 s, 4.4 GB/s
+ dd if=/mnt/fest/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.229069 s, 4.7 GB/s

I have a few questions:

1: Why is read performance so horrendously bad at 849 MB/s when I read the file back via glusterfs after I have just read the file from the local filesystem at 5.3 GB/s and thus ensuring that the file is in page cache for the glusterfs server to read from?

2: When I write the file locally and read it locally I can read the file out of page cache at more than 4 GB/s the first time I read it as well as the second, but when I write the file via glusterfs and read it locally the first read is much slower, indicating that the glusterfs server does something strange wrt. caching, what's going on here?

I'm aiming to run the real system on 40Gb/s IB, so I'm a bit annoyed that glusterfs seems to bottleneck at 22% of line speed even on localhost, without any real IO going on.

--

Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users