To be fair, I'm not even sure it's technically a "leak", it's just io-
cache not limiting itself to the cache-size setting. As in, io-cache
is doing its job, it's just doing it a bit .. too enthusiastically.
First, here's my io-cache block:
volume ioc
type performance/io-cache
subvolumes stripe0 # In this example it is 'client' you may
have to change it according to your spec file.
option page-size 1MB # 128KB is default
option cache-size 2048MB # 32MB is default
option force-revalidate-timeout 5 # 1second is default
option priority *.psiblast:3,*.seq:2,*:1
end-volume
And for my test:
[root@cpu26 ~]# cd /distfs
[root@cpu26 distfs]# ls -lah bigfile.img
-rw-r--r-- 1 root employees 6.3G Aug 26 00:21 bigfile.img
Note the memory usage of a freshly started glusterfs client:
root 23972 0.0 0.0 110340 1540 ? Ssl 11:37 0:00
[glusterfs]
Now I do this:
[root@cpu26 distfs]# cat bigfile.img > /dev/null
The glusterfs memory footprint grows at a nice rate. In just a few
seconds it's gone from 110kb to 2.5GB, way past my cache-size limit.
root 23972 9.4 61.8 2590552 2503164 ? Ssl 11:37 0:14
[glusterfs]
This box has only 4GB RAM so i killed the 'cat' process before things
got out of hand. But, there's your test.
[root@cpu26 ~]# glusterfs --version
glusterfs 1.3.11 built on Aug 21 2008 11:26:38
Repository revision: glusterfs--mainline--2.5--patch-795
Dan Parsons
On Nov 7, 2008, at 11:31 AM, Krishna Srinivas wrote:
Lukas, Dan,
Are you sure that its a leak of io-cache? did you try by removing the
translator and not observe the leak?
What made you conclude that cache-size option was being ignored? It
could be a memory leak by io-cache at some other place too.
I tried to make io-cache memleak, but its not happening, what
operations do you guys do to see the leak? can you see if some simple
test case makes it leak? (so that it will be easy for me to reproduce
the bug here and fix it)
Thanks!
Krishna
On Wed, Nov 5, 2008 at 11:38 PM, Dan Parsons <dparsons@xxxxxxxx>
wrote:
Lukas, just to confirm your findings, I have the exact same problem
and
reported it about 2 months ago. Just like you, when all my stuff
was running
under 32-bit, it wasn't an issue because of the 2GB limit, but now
that I'm
using 64-bit for everything, it is a potential system crashing bug.
Dan Parsons
On Nov 5, 2008, at 1:05 AM, Lukas Hejtmanek wrote:
Hello,
On Tue, Nov 04, 2008 at 12:37:03PM +0530, Krishna Srinivas wrote:
We want to reproduce the leak in our setup to fix it. What is your
setup on the client side? How many servers do you have? What are
the
applications you run on the mount point? Do you observe leak only
when
"certain" operations are done? (I am just looking for more clues)
upon my investigation, it seems that the io-cache translator
ignores the
cache
size config option and on 64bit systems is can easily eat whole
memory. On
32bit, it usually fails to mmap additional data due to 2GB limit to
process
address space.
--
Lukáš Hejtmánek
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
