Hey all, My company has gone live with a GFS cluster this morning. It is a 4 node RHEL4U6 cluster, running RHCS and GFS. It mounts an Apple 4.5TB XRAID configured as RAID5, whose physical volumes are combined into one large volume group. From this volume group, five striped LVMs (striped across the two physical volumes of the XRAID) were created. Five GFS filesystems were created, one on each logical volume. Even though there are currently four nodes, there are 12 journals for each filesystem to allow for planned cluster growth. Currently, each filesystem is mounted noatime, and tunebles quota_enforce and quota_account are set to 0. I have posted the results of gfs_tool gettune /hq-san/nlp/nlp_qa below. We have an application which depends heavily upon a find command that lists a number of files. It looks something like this: find $delta_home -name summary -maxdepth 2 -type f Its output consists of thousands of files that exist on /hq-san/nlp/nlp_qa. This command is CRAWLING at the moment. An ext3 filesystem would output hundreds of matches a second. This GFS filesystem is currently outputting 100-200/minutes. This is crippling one of our applications. Any advice on tuning this filesystem for this kind of access would be greatly appreciated. Output from a gfs_tool df /hq-san/nlp_qa: odin / # gfs_tool df /hq-san/nlp/nlp_qa /hq-san/nlp/nlp_qa: SB lock proto = "lock_dlm" SB lock table = "boson:nlp_qa" SB ondisk format = 1309 SB multihost format = 1401 Block size = 4096 Journals = 12 Resource Groups = 3009 Mounted lock proto = "lock_dlm" Mounted lock table = "boson:nlp_qa" Mounted host data = "" Journal number = 1 Lock module flags = Local flocks = FALSE Local caching = FALSE Oopses OK = FALSE Type Total Used Free use% ------------------------------------------------------------------------ inodes 15167101 15167101 0 100% metadata 868298 750012 118286 86% data 219476789 192088469 27388320 88% Output from a df -h: /dev/mapper/hq--san-cam_development 499G 201G 298G 41% /hq-san/nlp/cam_development /dev/mapper/hq--san-nlp_qa 899G 794G 105G 89% /hq-san/nlp/nlp_qa /dev/mapper/hq--san-svn_users 1.5T 1.3T 282G 82% /hq-san/nlp/svn_users /dev/mapper/hq--san-development 499G 373G 126G 75% /hq-san/nlp/development /dev/mapper/hq--san-prod_reports 1023G 680G 343G 67% /hq-san/nlp/prod_reports odin / # gfs_tool gettune /hq-san/nlp/nlp_qa ilimit1 = 100 ilimit1_tries = 3 ilimit1_min = 1 ilimit2 = 500 ilimit2_tries = 10 ilimit2_min = 3 demote_secs = 300 incore_log_blocks = 1024 jindex_refresh_secs = 60 depend_secs = 60 scand_secs = 5 recoverd_secs = 60 logd_secs = 1 quotad_secs = 5 inoded_secs = 15 glock_purge = 0 quota_simul_sync = 64 quota_warn_period = 10 atime_quantum = 3600 quota_quantum = 60 quota_scale = 1.0000 (1, 1) quota_enforce = 0 quota_account = 0 new_files_jdata = 0 new_files_directio = 0 max_atomic_write = 4194304 max_readahead = 262144 lockdump_size = 131072 stall_secs = 600 complain_secs = 10 reclaim_limit = 5000 entries_per_readdir = 32 prefetch_secs = 10 statfs_slots = 64 max_mhc = 10000 greedy_default = 100 greedy_quantum = 25 greedy_max = 250 rgrp_try_threshold = 100 statfs_fast = 0 seq_readahead = 0 Shawn -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
