I have optimized my glusterfs system pretty well for writes, but I am having an issue where doing an ls is very slow, and during I/O operations df takes forever and often hangs on glusterfs. Below are some examples of the slow ls (at the time of the tests, there are no filesystem operations occuring other than the ls). I am finding that during heavy reads problems are much worse, though they also can be bad during heavy writes. Does anyone have any ideas of how I can track this problem down? It is the only thing preventing our system from being production-ready. The current hardware setup is 2 storage servers using ALU on a 16xSATA RAID6 (8cores each w/ 8gb RAM), and an AFR'ed namespace on the same hardware RAIDs. My configuration is available at http://genome.ucla.edu/~jordan/gluster/ Thanks so much, Jordan ----------------- Examples of slow ls during no FS operations: [root@solexa solexa]# time ls solexa_datasets/ 070403_SLXA-EAS20_0001_FC5063 071121_HWI-EAS172_200C0 docs 070405_SLXA-EAS20_0005_FC5065 071203_HWI-EAS172_200BA genomes 070418_SLXA-EAS20_5256 071207_HWI-EAS172_2015A Instruments 070427_SLXA-EAS20_5152 071211_HWI-EAS172_200gh L001.tar 070515_SLXA-EAS20_5612 071217_HWI-EAS172_14759 L007.tar 070523_SLXA-EAS20_5153 071221_HWI-EAS172_200M1 ls_d_images.output 070529_SLXA-EAS20_5594 080108_HWI-EAS172_200G8 ls_d_images.output.20071029 070604_SLXA-EAS20_5447 080116_HWI-EAS172_14758 ls_d_images.output.20071029.truncated 070608_SLXA-EAS20_FC5459 080121_HWI-EAS172_2040F ls_d_images.output.2007103016.truncated 070612_SLXA-EAS20_5646 080125_HWI-EAS172_13471R ls_d_images.output.2007103017.truncated 070621_SLXA-EAS20_5590 080128_HWI-EAS172_201UD ls_d_images.output.20071030.truncated 070625_SLXA-EAS20_5701 080204_HWI-EAS172_201ET ls_d_images.output.200710311329.truncated 070629_SLXA-EAS20_5861 080206_SLXA-EAS20_203NA ls_d_images.output.200710311349.truncated 070703_SLXA-EAS20_5731 080215_HWI-EAS172_2009B ls_d_images.output.200710311350.nottruncatedbutmissingdoneitems 070709_SLXA-EAS20_5611 080226_HWI-EAS172_205A0 ls_d_images.output.200710311749.truncated 070713_SLXA-EAS20_5863 080304_HWI-EAS172_204AY ls_d_images.output.200711010821.truncated 070717_SLXA-EAS20_5699 080307_HWI-EAS172_20ANM ls_d_images.output.200711010845.truncated 070723_SLXA-EAS20_5540 080311_HWI-EAS172_204KJ ls_d_images.output.200711051802.truncated 070727_SLXA-EAS20_4606 080314_HWI-EAS172_204MG ls_d_images.output.clustertemp1.200711011622 070731_SLXA-EAS20_4611 080320_HWI-EAS172_204KB redo.200711061339 070810_SLXA-EAS20_5866 080326_HWI-EAS172_204KP Reports 070814_SLXA-EAS20_5697 080401_HWI-EAS172_20CLK rsync_to_storage.pl 070817_SLXA-EAS20_5603 080407_HWI-EAS172_204MN Runs_Comments 070821_SLXA-EAS20_11290 080411_HWI-EAS172_204K1 Shawn_basecalls_U87 070827_SLXA-EAS20_11279 080418_HWI-EAS172_204T7 solexatmp0.tarfile2dirname 070831_SLXA-EAS20_5406 080429_HWI-EAS172_204KD solexatmp0.tarfile2dirname.new 070925_SLXA-EAS20_11296 080505_HWI-EAS172_20DYC solexatmp1.tarfile2dirname 071002_SLXA-EAS20_4977 080513_HWI-EAS172_20DYG testdir 071022_HWI-EAS172_11989 080515_HWI-EAS172_20DYG_RE testdir.tar 071026_HWI-EAS172_11989reseq 080516_HWI-EAS335_305T4 test.pl 071030_HWI-EAS172_14517 1 U87.fasta.bz2 071105_HWI-EAS172_14515 4276.fasta verify_JMM.log 071109_HWI-EAS172_14034 -al verify_JMM.sh 071115_HWI-EAS172_14055 clustertemp1.tarfiles.200711051731 real 0m2.298s user 0m0.001s sys 0m0.001s [root@solexa solexa]# time ls solexa_datasets/070817_SLXA-EAS20_5603/Data C2-36_Firecrest1.8.28_04-10-2007_solexa default_offsets.txt real 0m6.246s user 0m0.001s sys 0m0.002s
