On Tue, Jun 17, 2008 at 5:22 PM, Terry <td3201@xxxxxxxxx> wrote: > On Tue, Jun 17, 2008 at 3:09 PM, Wendy Cheng <s.wendy.cheng@xxxxxxxxx> wrote: >> Hi, Terry, >>> >>> I am still seeing some high load averages. Here is an example of a >>> gfs configuration. I left statfs_fast off as it would not apply to >>> one of my volumes for an unknown reason. Not sure that would have >>> helped anyways. I do, however, feel that reducing scand_secs helped a >>> little: >>> >> >> Sorry I missed scand_secs (was mindless as the brain was mostly occupied by >> day time work). >> >> To simplify the view, glock states include exclusive (write), share (read), >> and not-locked (in reality, there are more). Exclusive lock has to be >> demoted (demote_secs) to share, then to not-locked (another demote_secs) >> before it is scanned (every scand_secs) to get added into reclaim list where >> it can be purged. Between exclusive and share state transition, the file >> contents need to get flushed to disk (to keep file content cluster >> coherent). All of above assume the file (protected by this glock) is not >> accessed (idle). >> >> You hit an area that GFS normally doesn't perform well. With GFS1 in >> maintenance mode while GFS2 seems to be so far away, ext3 could be a better >> answer. However, before switching, do make sure to test it thoroughly (since >> Ext3 could have the very same issue as well - check out: >> http://marc.info/?l=linux-nfs&m=121362947909974&w=2 ). >> >> Did you look (and test) GFS "nolock" protocol (for single node GFS)? It >> bypasses some locking overhead and can be switched to DLM in the future >> (just make sure you reserve enough journal space - the rule of thumb is one >> journal per node and know how many nodes you plan to have in the future). >> >> -- Wendy > > Good points. I could try the nolock feature I suppose. Not quite > clear on how to reserve journal space. I forgot to post the cpu time, > check out this: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4822 root 10 -5 0 0 0 S 1 0.0 2159:15 dlm_recv > 4820 root 10 -5 0 0 0 S 1 0.0 368:09.34 dlm_astd > 4821 root 10 -5 0 0 0 S 0 0.0 153:06.80 dlm_scand > 3659 root 10 -5 0 0 0 S 0 0.0 134:40.14 scsi_wq_4 > 4823 root 11 -5 0 0 0 S 1 0.0 109:33.33 dlm_send > 367 root 10 -5 0 0 0 S 0 0.0 103:33.74 kswapd0 > > gfs_glockd is further below so not so concerned with that right now. > It appears turning on nolock would do the trick. The times aren't > extremely accurate because I have failed this cluster between nodes > while testing. > Here is some more testing information.... I created a new volume on my iscsi san of 1 TB and formatted it for ext3. I then used dd to create a 100G file. This yielded roughly 900 Mb/sec. I then stopped my application and did the same thing with an existing GFS volume. This gave me about 850 Kb/sec. This isn't an iscsi issue. This appears to be a load issue and the number of I/O occurring on these volumes. That said, I would expect that performing the changes I did would result in a major performance improvement. Since it didn't, what are my other points I could consider? If its a GFS issue, ext3 is the way to go. Maybe even switch to using active-active on my NFS cluster. If its a backend disk issue, I would expect to see the throughput on my iscsi link (bond1) be fully utilized. Its not. Could I be thrashing the disks? This is an iscsi san with 30 sata disks. Just bouncing some thoughts around to see if anyone has any more thoughts. Thanks! -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
