Hi All, I'm experiencing extremely slow auto-heal rate with glusterfs and I want to hear from you guys to see if it seems reasonable or something's wrong. I did a reconfiguration of the glusterfs running on our lab cluster. Originally we have 25 nodes, each provide 1 1.5T SATA disk, the data are not replicated. Now we have expanded the cluster to 66 nodes and I decided to use all the 4 disks of each node and have the data replicated 3 times on different machines. What I did is to leave the original data untouched, and reconfigure glusterfs so that the original data volumes are paired up with new empty mirror volumes. On the server side, I have each node exports one volume for each of the 4 disks, with an IO-threads translator with 4 threads running on top of each disk and no other performance translators. On the client side which is mounted on each of the 66 nodes, I group all the volumes into mirrors of 3 volumes each, and then aggregate the mirrors with one DHT, and put a write-behind translator with window-size 1MB on top of that. The files are all images, roughly 200K each. To trigger auto-heal, I split a list of all the files (previously saved before reconfiguration) among the 66 nodes and have 4 threads on each nodes running the stat command on the files. The overall rate is about 50 files per second, which I think is very low. And will the auto-heal is running, all operations like cd and ls on the glusterfs client becomes extremely slow, each takes like one minute to finish. On http://www.gluster.org/docs/index.php/GlusterFS_2.0_I/O_Benchmark_Results, which uses 5 servers and one raid0 disk each node, with 5 threads running on 3 clients, about 61K 1M files can be created within 10min, at an average rate of 130 files/second. The glusterfsd processes are each taking about 99% CPU time ( we have 8 cores / node, but it seems glusterfsd is able to use only 1). There only advantage is that they use RAID0 disk and clients and servers are different machines. Other than that, we have more servers/clients, more disks on each node, and I have configured IO-thread and write-behind (which I don't think helps auto heal), and our files are only about 1/5 their size. Even if I count each file as 3 for replication, I'm only achieving similar throughput as a much smaller cluster. I don't know why it's like this and following are my hypothesis: 1. The overall through of glusterfs doesn't scale up to so many nodes; 2. The auto-heal operating is slower than creating the file from scratch; 3. Glusterfs is CPU bound -- it seems to be the case, but I'm unwilling to accept that a filesystem is CPU bound; 4. The network bandwith is saturated. I think we have 1gigabit eithernet on the nodes, which are on two racks and connected by some Foundry switch with 4 gigabit aggregate inter-rack bandwidth; 5. Replication is slow. Finally, the good news is that all the servers and client processes have been running for about 2 hours under stress, which I didn't expect in the beginning. Good job you glusterfs people! We are desperate for a shared storage and I'm eager to hear any suggestion you have to make glusterfs perform better. - Wei
