Hi everyone, I hope you can help me with some performance troubles I've been having. I'm doing some tests with gluster 3.2.0, and I can't understand some of the behavior I'm getting with gluster. The test is using a volume with 12 stripped bricks (each brick is an HD), with no replication, via RDMA. I'm doing random readings of 4GByte files with the FIO tool. Gluster reading speed is of 150-160MByte/s and 110-120MByte/s per client if I connect with two clients. However, on the same machine, I have a similar lustre setup with a constant 400MByte/s throughput, for the same tests, with the same amount of disks going through the same RAID controller. Both tests were ran at different times. As comparison, for the sequential reading tests, both gluster and lustre give me, respectively, results of 600 to 700MByte/s. The gluster configuration files have no extra modifications, the disks are formatted with ext3 and created via: gluster volume create test stripe 16 transport rdma 10.1.0.4:/disk1 10.1.0.4:/disk2 ... 10.1.0.4:/disk16 Using DD to write/read from each of the disks gives me about 100MByte/s. Would there be anything obvious I'm missing? What would be causing this low reading speed? I tried playing with the performance parameters, setting up more or less cache and io-threads, but the results did not improve. Moreover, I have tested all the suggested optimization hacks/parameters with little change in the results. A little more insight on the setup: we have 24 HD machines, with dual Xeon CPUs, 16GByte of RAM, working over an Areca RAID card, using JBOD, all boxes connected via 10Gbit/s Infiniband. The CPUs are far from being maxed out (top reports around 50% idle), as is the network. I'm using CentOS 5.5 on all the machines. Let me know if you need more information on what's going on (configuration files, setup, etc, etc). Thanks in advance, Daniel ????????????????????????? ?SkillUpJapan Corporation ?Research and Development Office Senior Researcher Engineer ?Daniel Pereira?d.pereira at skillupjapan.co.jp ?Tokyo, Shinjuku, Takadanobaba 1-24-16 Uchida Building 1st Floor ?TEL:03-5287-4087?FAX:03-5287-4135 http://www.skillupjapan.co.jp/ ?????????????????????????
