Chris Joelly wrote:
Hello,
i have a question on locking issues on GFS:
how do GFS lock files on the filesystem. I have found one posting to
this list which states that locking occurs "more or less" on file
level. Is this true? or does some kind of locking occur on directory
level too?
You may view GFS(1) internal lock granularity is on system call level -
that is, when either a write or read (say pwrite() or pread()) is
issued, the associated file is locked until the system call returns.
There are few simple things that will be helpful if you keep them in mind:
1. a write requires an exclusive lock (i.e., when there is a write going
on, every access to that file has to wait).
2. a read needs a shared lock (i.e. many reads to the same file will
not be stalled).
3. a write may involve directory lock (e.g. a "create" would need a
write lock of the parent directory).
4. local locking (two writes compete the same lock on the same node) is
always much better than inter-node (different nodes) locking (ping-pong
the same write lock between different nodes is very expensive).
5. standard APIs (such as fcntl() and flock()) precedes GFS(1) internal
locking if used correctly (e.g. upon obtaining an exclusive flock, other
access to that file will be stalled, assuming every instance of the
executables running on different nodes has the correct flock implemented
and honored).
One exception to (5) is posix byte-range locks. Say there are two
processes running on different nodes, each obtaining its own byte range
locks. Process A locks byte offset 0 to 10K; process B locks byte 10K+1
to 40K. When both have writes issued, one of them has to wait until
other's write system call completes before it can continue - a result of
its posix locking implementation that is done by a separate module
outside its internal filesystem locking.
The question arrose because i was thinking on whats would happen when
using GFS on an file server with home directories for lets say 1000
users. how do i setup this directory structure to avoid locking issues
as best as possible.
is a directory layout like the following ok when /home is a GFS file
system:
/home/a/b/c/abc
/home/d/e/f/def
/home/g/h/i/ghi
...
/home/x/y/z/xyz
Hope above statements have helped you understanding that on GFS(1),
1. A short-and-fat directory structure will work (much) better than
tall-and-skinny ones.
2. If possible, the directory setup should avoid ping-pong directory
and/or write locks between different nodes
For GFS2, after browsing thru its newest source code few minutes ago -
it reminds me of a "sea horse" shape with linux page locks on its curly
"belly" :). It is difficult (for me) to describe so I'll skip.
-- Wendy
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
