On Thu, Aug 26, 2004 at 10:55:29AM -0700, Cahill, Ben M wrote: > Can anyone (Ken?) explain why num_glockd mount > option is there? TIA. EOM. One of the things (probably the major thing) that GFS uses memory for that's different from other filesystems is locks. That introduces the interesting property that in order for GFS to free that memory, it has to do network I/O to unlock locks. (The same is true for memory that contains dirty disk blocks, but the VM knows about that.) This means that freeing memory can only happen so quickly. In the past, GFS had one thread (gfs_glockd) that would scan through the glock hash table looking for cached locks that were no longer needed. It would then unlock those locks and free the memory associated with them. But as it turned out, there were memory problems, if you had a few processes that were scanning through huge directory trees. You could get into a situation where you were acquiring new locks much faster than gfs_glockd could release old ones. (There were many threads acquiring, but only one thread releasing.) If this kept up, you'd soon run out of memory and bad things would happen. My first quick-n-dirty solution to this was to add the num_glockd mount option. It would create many many threads that would look for unused locks and unlock them. You could balance out acquiring and releasing processes if you knew what workload you were running. So, that's why the option was there originally. Multiple gfs_glockd processes didn't completely solve the problem, though. You could still get into the situation where GFS wasn't responding quickly enough to memory pressure. So I made a bunch of changes that made things a lot better: 1) I broke gfs_glockd into two threads: A) gfs_scand - scans the glock hash table looking for glocks to demote. When it finds one, it puts it onto a reclaim list of unneeded locks, and wakes up gfs_glockd. B) gfs_glockd - looks at that the glocks on that reclaim list and starts demoting them. 2) When the number of locks on the reclaim list becomes too great, threads that want to acquire new locks will pitch in and release a couple of locks before acquiring a new one. 3) GFS is more proactive about putting locks that it knows it won't need onto the reclaim list. This reduces the need for gfs_scand and actually walking the hash table. There's more work that could be done in this area. So, I left the num_glockd option there to in case it's still needed for some reason. But because of #2 above, I don't think it will be. The option may go away in the future. -- Ken Preslan <kpreslan@xxxxxxxxxx>
