Interesting. Can you share more details of your application, such as the access patterns of the files/directories involved, and the directory layout? We faced some similar challenges in our cluster, in which we use GFS for web session storage. Performance was improved dramatically by several changes we experimented with: - GFS tunables: glock_purge, demote_secs, etc.) - Network optimization: using different interfaces tuned for DLM/OpenAIS traffic vs. iSCSI - Locality: Having each node create session files in a directory dedicated to that node. Different nodes can read files created on other nodes, but are not permitted to create files outside their own directory. - Session affinity: Having repeat session hits come back to the same node when possible. We also found we needed to specify "nodiratime" in addition to "noatime", though this contradicts certain documentation that suggests the two are redundant. GFS2 may significantly outpeform GFS for these usage patterns. As RHEL 5.3 is still in beta and the implementation of GFS2 in 5.2 is not production-ready, we have not had much opportunity to test GFS2 ourselves. Some general hints we found: - If you have much temporal locality in your application, tune GFS to extend the lifetime of locks, such as increasing demote_secs. Consecutive accesses by the same node to the same file over GFS may perform nearly as well as local storage. - Avoid having different nodes create files in a common directory at once, if possible. This creates a ping-pong effect as the directory lock bounces between nodes. - If you have a particularly large number of files, try setting glock_purge to a value greater than zero. This may help especially if you are experiencing high system CPU usage. - Pay attention to statistics such as you'll see with "gfs_tool counters" to understand how many locks are acquired by your app over time, and how often they are accessed. Any change to decrease the frequency of lock traffic is a performance win. GFS can be a bit of a fuss to configure, but once running our experience with 5.2 and GFS has been rock-steady. Good luck. Jeff > -----Original Message----- > From: linux-cluster-bounces@xxxxxxxxxx > [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Shaun Mccullagh > Sent: Monday, December 08, 2008 3:00 PM > To: linux clustering > Subject: GFS on CentOS 5.2 > > > Hi, > > A few weeks ago I started to setup a 6 node GFS cluster > connected to a SAN with Fiber HBAs. Each node is connected to > 2 gigabit switches for redundancy. > > Up until a few days ago things were going very well, but > intensive testing showed that we have a bit of a performance problem. > > Perhaps I should explain that we want to use GFS in a LAMP > environment to speed up a customer website. > > The old system consisted of six nodes which shared files > using glorious NFS. > > But we have found to our dismay that Apache is taking much > longer to access files via GFS than NFS. Strace shows that > the Apache workers are waiting on fstat calls. They take 4 > times longer to complete on GFS than NFS > > I have tried the following: > > - disabled atime updating: mount option 'noatime' > - disabled quota: mount option 'noquota' > - increased statfs_slots: gfs_tool settune <mountpoint> > statfs_slots 128 > - enabled statfs_fast (gfs_tool settune <mountpoint> statfs_fast 1) > > These improved things slightly but GFS is still much slower than NFS > > The CPU load is much higher with GFS, vmstat shows that > Context switching is > 5 greater than that of an NFS client. > > The GFS filesystem is only 10GB. > > We are getting a bit desperate so I'd grateful for any > pointers as to what might be going wrong. > > TIA > > Shaun > > Op dit e-mailbericht is een disclaimer van toepassing, welke > te vinden is op http://www.espritxb.nl/disclaimer > NB: Vanaf heden zijn al onze mailadressen gewijzigd in > ...@xxxxxxxxxxx! Pas dit svp aan in uw adresboek. > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > > -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
