Hi, I had (nearly) the same problem. A slow gfs. From the beginning. Two weeks ago the cluster crashed every time the load became heavier. What was the reason? A rotten gfs. The gfs uses leafnodes for data an leafnodes for metadata whithin the filesystem. And the problem was in the metadata leafnodes. Have you checked the Filesystem? Unmount it from all nodes and use gfs_fsck on the filesystem. In my case it reported (and repaired) tons of unused leafnoedes and some other errors. First time I started it without the -y (for yes). Well, after one hour ot typing y I killed it and started it with -y. The work was done whithin an hour for 1TB. Now the filesystem is clean and it was like a turboloader and Nitrogen injection for a car. Fast as it was never before. Maybe there is a bug in the mkfs command or so. I will never use a gfs without a filesystem check after creation Martin Fuerstenau Seniro System Engineer Oce Printing Systems, Poing On Fri, 2008-05-09 at 02:25 -0700, Ja S wrote: > Hi, Klaus: > > Thank you very much for your kind answer. > > Tunning the parameters sounds really interesting. I > should give it a try. > > By the way, how did you come up with these new > parameter values? Did you calculate them based on > some measures or simply pick them up and test. > > Best, > > Jas > > > --- Klaus Steinberger > <Klaus.Steinberger@xxxxxxxxxxxxxxxxxxxxxx> wrote: > > > Hi, > > > > > However, it took ages to list the subdirectory on > > an > > > absolute idle cluster node. See below: > > > > > > # time ls -la | wc -l > > > 31767 > > > > > > real 3m5.249s > > > user 0m0.628s > > > sys 0m5.137s > > > > > > There are about 3 minutes spent on somewhere. Does > > > anyone have any clue what the system was waiting > > for? > > > > Did you tune glock's? I found that it's very > > important for performance of > > GFS. > > > > I'm doing the following tunings currently: > > > > gfs_tool settune /export/data/etp quota_account 0 > > gfs_tool settune /export/data/etp glock_purge 50 > > gfs_tool settune /export/data/etp demote_secs 200 > > gfs_tool settune /export/data/etp statfs_fast 1 > > > > Switch off quota off course only if you don't need > > it. All this tunings have > > to be done every time after mounting, so do it in a > > init.d script running > > after GFS mount, and of course do it on every node. > > > > Here is the link to the glock paper: > > > > > http://people.redhat.com/wcheng/Patches/GFS/readme.gfs_glock_trimming.R4 > > > > The glock tuning (glock_purge and demote_secs > > parameters) definitly solved a > > problem we had here with the Tivoli Backup Client. > > Before it was running for > > days and sometimes even did give up. We observed > > heavy lock traffic. > > > > After changing the glock parameters times for the > > backup did go down > > dramatically, we now can run a Incremental Backup on > > a 4 TByte filesystem in > > under 4 hours. So give it a try. > > > > There is some more tuning, which could be done > > unfortunately just on creation > > of filesystem. The default number of Resource Groups > > is ways too large for > > nowadays TByte Filesystems. > > > > Sincerly, > > Klaus > > > > > > -- > > Klaus Steinberger Beschleunigerlaboratorium > > Phone: (+49 89)289 14287 Am Coulombwall 6, D-85748 > > Garching, Germany > > FAX: (+49 89)289 14280 EMail: > > Klaus.Steinberger@xxxxxxxxxxxxxxxxxxxxxx > > URL: > > > http://www.physik.uni-muenchen.de/~Klaus.Steinberger/ > > > -- > > Linux-cluster mailing list > > Linux-cluster@xxxxxxxxxx > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > ____________________________________________________________________________________ > Be a better friend, newshound, and > know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > Martin Fürstenau Tel. : (49) 8121-72-4684 Oce Printing Systems Fax : (49) 8121-72-4996 OI-12 E-Mail : martin.fuerstenau@xxxxxxx Siemensallee 2 85586 Poing Germany Visit Oce at drupa! Register online now: <http://drupa.oce.com> This message and attachment(s) are intended solely for use by the addressee and may contain information that is privileged, confidential or otherwise exempt from disclosure under applicable law. If you are not the intended recipient or agent thereof responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by telephone and with a 'reply' message. Thank you for your co-operation. -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
