Wendy Cheng wrote:
Chris Joelly wrote:
Hello,
i have a question on locking issues on GFS:
how do GFS lock files on the filesystem. I have found one posting to
this list which states that locking occurs "more or less" on file
level. Is this true? or does some kind of locking occur on directory
level too?
You may view GFS(1) internal lock granularity is on system call level
- that is, when either a write or read (say pwrite() or pread()) is
issued, the associated file is locked until the system call returns.
There are few simple things that will be helpful if you keep them in
mind:
1. a write requires an exclusive lock (i.e., when there is a write
going on, every access to that file has to wait).
2. a read needs a shared lock (i.e. many reads to the same file will
not be stalled).
3. a write may involve directory lock (e.g. a "create" would need a
write lock of the parent directory).
4. local locking (two writes compete the same lock on the same node)
is always much better than inter-node (different nodes) locking
(ping-pong the same write lock between different nodes is very
expensive).
5. standard APIs (such as fcntl() and flock()) precedes GFS(1)
internal locking if used correctly (e.g. upon obtaining an exclusive
flock, other access to that file will be stalled, assuming every
instance of the executables running on different nodes has the correct
flock implemented and honored).
One exception to (5) is posix byte-range locks. Say there are two
processes running on different nodes, each obtaining its own byte
range locks. Process A locks byte offset 0 to 10K; process B locks
byte 10K+1 to 40K. When both have writes issued, one of them has to
wait until other's write system call completes before it can continue
- a result of its posix locking implementation that is done by a
separate module outside its internal filesystem locking.
The question arrose because i was thinking on whats would happen when
using GFS on an file server with home directories for lets say 1000
users. how do i setup this directory structure to avoid locking issues
as best as possible.
is a directory layout like the following ok when /home is a GFS file
system:
/home/a/b/c/abc
/home/d/e/f/def
/home/g/h/i/ghi
...
/home/x/y/z/xyz
Hope above statements have helped you understanding that on GFS(1),
1. A short-and-fat directory structure will work (much) better than
tall-and-skinny ones.
2. If possible, the directory setup should avoid ping-pong directory
and/or write locks between different nodes
Really think about it, (1) is *not* a right description at all. What I
meant to say is that, if possible, try not to put everyone in the very
same directory if there are lots of write activities that will cause
directory lock(s) contention, particularly if it (they) has (have) to
get passed around different nodes.
Sorry !
-- Wendy
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
