Re: io-cache exceeding cache-size value

[Dan Parsons <dparsons@xxxxxxxx>]

I will do this today. I noticed that I already have vm.drop_caches set to 3 via sysctl.conf, based on a suggestion from you from long ago. Should I delete this under normal usage? Is it possible that this setting, enabled by default, is causing my problems?

Dan

On Mon, Feb 23, 2009 at 3:23 PM, Amar Tumballi (bulde) <amar@xxxxxxxxxxx> wrote:

Hi Gordan and Dan,

It would help me a lot if its possible for you to get the info as described below,

compile glusterfs like

bash# make clean > /dev/null
bash# make CFLAGS="-g -O0 -DDEBUG" > /dev/null
bash# make install

run the process which consumes memory (mostly client process) like below:

bash# glusterfs <any argument you give generally> -N
<this process will run in foreground now>

Open another terminal

bash# ps aux | grep glusterfs
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check in other terminal for memory usage stats>

bash# <run your application over glusterfs as you do till you get high memory usage of glusterfs.. >
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check the stat in another terminal>

bash# echo 3 > /proc/sys/vm/drop_caches
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check the stat in another terminal>

Even after dropping caches, if 'in use bytes =' in malloc stats shows a high value, then it is a leak. If its showing less, but just 'system bytes = ' is a high value, this means glusterfs is not really consuming high memory, but the problem is really in the memory allocation segments.

Regards,
Amar

NOTE: 'malloc_stats' will be printed to 'stdout' if we enable -DDEBUG while compiling glusterfs, as it hits performance badly otherwise.

2009/2/23 Gordan Bobic <gordan@xxxxxxxxxx>

Dan Parsons wrote:

I'm having an issue with glusterfs exceeding its cache-size value. Right now I have it set to 4000MB and I've seen it climb as high as 4800MB. If I set it to 5000, I've seen it go as high as 6000MB. This is a problem because it causes me to set the value very low so that my apps don't get pushed into swap. Is there any way to fix this? To get it to stick to the limit I set and not exceed?

It's possible you are running into the same memory leak that I'm seeing, and I'm not using io-cache or any other performance translators at all. With rootfs on Gluster, doing a kernel compile (kernel source tree being on NFS, so this won't be contributing to the bloat, hopefully) makes the glusterfsd bloat by about 80MB per pass, and never frees it.

Gordan


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel

--
Amar Tumballi
Gluster/GlusterFS Hacker
[bulde on #gluster/irc.gnu.org]
http://www.zresearch.com - Commoditizing Super Storage!

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel