thanks for reporting the observation with fcntl locks. There was a memory leak in posix-locks which was fixed in 1.4 tree, but had somehow missed the backport to 1.3 tree. Please find the memory leak fix in the latest revision of glusterfs--mainline--2.5.
thanks,
avati
2008/11/4 Thomas Conway-Poulsen <tecp@xxxxxxxxxxx>
Hi Krishna
Running with fuse-2.7.3glfs10.
The client setup are as follows:
------------------------------------------------------------------------------------
mounts to /mnt/gluster/index:
------------------------------------------------------------------------------------
volume gluster01
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.1
option remote-subvolume gluster-storage-index
end-volume
volume gluster02
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.2
option remote-subvolume gluster-storage-index
end-volume
volume gluster03
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.3
option remote-subvolume gluster-storage-index
end-volume
volume gluster04
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.4
option remote-subvolume gluster-storage-index
end-volume
volume gluster05
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.5
option remote-subvolume gluster-storage-index
end-volume
volume gluster06
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.6
option remote-subvolume gluster-storage-indextype protocol/client
end-volume
volume gluster-storage-index-namespace
option transport-type tcp/client
option remote-host 10.11.253.1
option remote-subvolume gluster-storage-index-namespacetype cluster/unify
end-volume
volume gluster-storage-index
option scheduler rr
option namespace gluster-storage-index-namespace
subvolumes gluster01 gluster02 gluster03 gluster04 gluster05 gluster06subvolumes gluster-storage-index
end-volume
volume gluster-storage-index-iothreads
type performance/io-threads
option cache-size 64MB
option thread-count 8
end-volume
volume gluster-gluster-index-readahead
type performance/read-ahead
subvolumes gluster-storage-index-iothreads
end-volumevolume gluster-storage-index-iocache
volume gluster-storage-index-writebehind
type performance/write-behind
subvolumes gluster-storage-index-readahead
end-volume
type performance/io-cache
option cache-size 256MB------------------------------------------------------------------------------------
subvolumes gluster-storage-index-writebehind
end-volume
mounts to /mnt/gluster/data:
------------------------------------------------------------------------------------
volume cindex01
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.1
option remote-subvolume gluster-storage-data
end-volume
volume cindex02
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.2
option remote-subvolume gluster-storage-data
end-volume
volume cindex03
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.3
option remote-subvolume gluster-storage-data
end-volume
volume cindex04
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.4
option remote-subvolume gluster-storage-data
end-volume
volume cindex05
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.5
option remote-subvolume gluster-storage-data
end-volume
volume cindex06
type protocol/client
option transport-type tcp/client
option remote-host 10.11.253.6
option remote-subvolume gluster-storage-datatype protocol/client
end-volume
volume gluster-storage-data-namespace
option transport-type tcp/client
option remote-host 10.11.253.1
option remote-subvolume gluster-storage-data-namespacetype cluster/unify
end-volume
volume gluster-storage-data
option scheduler rr
option namespace gluster-storage-data-namespace
subvolumes cindex01 cindex02 cindex03 cindex04 cindex05 cindex06subvolumes gluster-storage-data
end-volume
volume gluster-storage-data-iothreads
type performance/io-threads
option cache-size 64MB
option thread-count 8
end-volume
volume gluster-storage-data-writebehind
type performance/write-behindsubvolumes gluster-storage-data-iothreadsvolume gluster-storage-data-iocache
end-volume
We can only see the memleak, when running multiple clients under heavy file-load over many files.
type performance/io-cache
option cache-size 128MB
subvolumes gluster-storage-data-writebehind
end-volume
The memory seems stable when we dont use fcnt locks-
bool lock(int file_fd) {
struct flock lock_info;
lock_info.l_type = F_RDLCK;
lock_info.l_whence = SEEK_SET;
lock_info.l_start = 0;
lock_info.l_len = 0;
if(fcntl(file_fd, F_SETLK, &lock_info) == 0) return true;
else return false;
}
bool unlock(int file_fd) {
struct flock lock_info;
lock_info.l_type = F_UNLCK;
lock_info.l_whence = SEEK_SET;
lock_info.l_start = 0;
lock_info.l_len = 0;
if(fcntl(file_fd, F_SETLKW, &lock_info) == 0) return true;
else return false;
}
bool lock(int file_fd, int start, int length) {
struct flock lock_info;
lock_info.l_type = F_RDLCK;
lock_info.l_whence = SEEK_SET;
lock_info.l_start = start;
lock_info.l_len = length;
if(fcntl(file_fd, F_SETLK, &lock_info) == 0) return true;
else return false;
}
bool unlock(int file_fd, int start, int length) {
struct flock lock_info;
lock_info.l_type = F_UNLCK;
lock_info.l_whence = SEEK_SET;
lock_info.l_start = start;
lock_info.l_len = length;
if(fcntl(file_fd, F_SETLKW, &lock_info) == 0) return true;
else return false;
}
Hope this helps,
Thomas Conway.
On Nov 4, 2008, at 8:07 AM, Krishna Srinivas wrote:
Thomas,
We want to reproduce the leak in our setup to fix it. What is your
setup on the client side? How many servers do you have? What are the
applications you run on the mount point? Do you observe leak only when
"certain" operations are done? (I am just looking for more clues)
Thanks
Krishna
On Sun, Nov 2, 2008 at 5:08 PM, Thomas Conway-Poulsen <tecp@xxxxxxxxxxx> wrote:
Hi devel,
The Glusterfsd process is using much more memory than we expected it to use,
maybe there is some memory leak?
It keeps eating until the process dies.
Is there any way to set the maximum memory usage ?
root 15436 1.5 85.5 2180580 1764980 ? Ssl Oct28 105:20 [glusterfs]
Our server configuration:
------------------------------------------------
volume gluster-storage-data-export
type storage/posix
option directory /mnt/gluster-storage-server/data/export
end-volume
volume gluster-storage-data-namespace
type storage/posix
option directory /mnt/gluster-storage-server/data/namespace
end-volume
volume gluster-storage-data-iothreads
type performance/io-threads
option thread-count 2
option cache-size 32MB
subvolumes gluster-storage-data-export
end-volume
volume gluster-storage-data-locks
type features/posix-locks
subvolumes gluster-storage-data-iothreads
end-volume
volume gluster-storage-data-readahead
type performance/read-ahead
subvolumes gluster-storage-data-locks
end-volume
volume gluster-storage-data-writebehind
type performance/write-behind
subvolumes gluster-storage-data-readahead
end-volume
volume gluster-storage-data
type performance/io-cache
option cache-size 128MB
subvolumes gluster-storage-data-writebehind
end-volume
volume gluster-storage-index-export
type storage/posix
option directory /mnt/gluster-storage-server/index/export
end-volume
volume gluster-storage-index-namespace
type storage/posix
option directory /mnt/gluster-storage-server/index/namespace
end-volume
volume gluster-storage-index-iothreads
type performance/io-threads
option thread-count 2
option cache-size 32MB
subvolumes gluster-storage-index-export
end-volume
volume gluster-storage-index-locks
type features/posix-locks
subvolumes gluster-storage-index-export
end-volume
volume gluster-storage-index-readahead
type performance/read-ahead
subvolumes gluster-storage-index-locks
end-volume
volume gluster-storage-index-writebehind
type performance/write-behind
subvolumes gluster-storage-index-readahead
end-volume
volume gluster-storage-index
type performance/io-cache
option cache-size 128MB
subvolumes gluster-storage-index-writebehind
end-volume
volume gluster-server
type protocol/server
subvolumes gluster-storage-index gluster-storage-index-namespace
gluster-storage-data gluster-storage-data-namespace
option transport-type tcp/server
option auth.ip.gluster-storage-index.allow *
option auth.ip.gluster-storage-index-namespace.allow *
option auth.ip.gluster-storage-data.allow *
option auth.ip.gluster-storage-data-namespace.allow *
end-volume
Here is the pmap dump:
-----------------------------------------------------
15436: [glusterfs]
Address Kbytes Mode Offset Device Mapping
0000000000400000 16 r-x-- 0000000000000000 008:00001 glusterfs
0000000000504000 4 rw--- 0000000000004000 008:00001 glusterfs
0000000000505000 497276 rw--- 0000000000505000 000:00000 [ anon ]
0000000040000000 4 ----- 0000000040000000 000:00000 [ anon ]
0000000040001000 8192 rw--- 0000000040001000 000:00000 [ anon ]
0000000040801000 4 ----- 0000000040801000 000:00000 [ anon ]
0000000040802000 8192 rw--- 0000000040802000 000:00000 [ anon ]
0000000041002000 4 ----- 0000000041002000 000:00000 [ anon ]
0000000041003000 8192 rw--- 0000000041003000 000:00000 [ anon ]
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
If I traveled to the end of the rainbow
As Dame Fortune did intend,
Murphy would be there to tell me
The pot's at the other end.
