Unexplicable GlusterFS load and poor performance

[Jelle Van Hees <jellev@xxxxxxxxxxxxxx>]

Dear gluster community

We are running GlusterFS replication server on 2 nodes and we have 2 clients. Self healing daemon is enabled. Each client is connected to a different server using the Gluster client. We have a lot of very small files on the Gluster volume.

We use GlusterFS 3.9.1 from the official GlusterFS APT Repositories on a Debian Jessie system.

The issue we are facing with is that one of the servers is at 0.1-0.5 load average while the other is at 200. The server with the high load also has a huge amount of data being streamed constantly to both the client nodes. This data stream continues even when the clients are not reading writing data.

The following values are measured by nload and iftop :

server: outgoing 35-40 MB/s

Two clients: incoming 17-20 MB/s

Our performance on the Gluster client is very poor. An ls can take up to 10 second to complete, and our app works extremely slow.

Servers and clients are connecting over an internal data-center network and can handle much more bandwidth so this is not a limiting factor.

My two main questions are :

1: Are these differences in server load normal behavior for GlusterFS and what causes this?

2: Why is there such a high constant data stream to clients form one of the servers.

I cannot seem to find any information concerning this in the Gluster Documentation or on the 

link to the server fault question

http://serverfault.com/questions/835343/unexplicable-glusterfs-load-and-poor-performance

Kind regards

JELLE VAN HEES LINUX SYSTEM ENGINEER AVIOVISION Jaarbeurslaan 19 box 11, 3600 Genk, Belgium T +32 (0)89 32 34 80 F +32 (0)89 36 37 95
Product of  AvioVision - A Thales Group Company

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, you are kindly requested to notify us immediately by replying to the message and deleting it from your computer.Any views or opinions presented in this email or any attachments are solely those of the author and do not necessarily represent those of AvioVision, or associated companies.

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users