Re: glusterfs brick server use too high memory

Nithya Balachandran <nbalacha@xxxxxxxxxx> · Mon, 13 Nov 2017 09:57:57 +0530

Hi,
It does look like the same problem. I would recommend upgrading and seeing if the issue goes away.

Thanks,
Nithya

On 9 November 2017 at 15:39, Yao Guotao <yaoguo_tao@xxxxxxx> wrote:
Thank you very much for your reply.

I think I found the key to the problem. But, I'd like you to confirm whether this is the issue.
My gluster version is 3.8.9, And I found there is a bug in it, and fixed in 3.8.11:
https://bugzilla.redhat.com/show_bug.cgi?id=1431592

A large amount of memory allocated for agf_common_mt_strdup. Everythig else seem to be all right.

My cluster:

Yesterday Afternoon：（I'm sorry I forgot the specific time.）
[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=1941483443
num_allocs=617382139
max_size=1941483443
max_num_allocs=617382139
total_allocs=661873332

TIme: 2017.11.9 17:15 (Today)
[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=792538295
num_allocs=752904534
max_size=792538295
max_num_allocs=752904534
total_allocs=800889589

The statedump files are in the attachment.

Thanks again.

在 2017-11-09 17:22:03，"Nithya Balachandran" <nbalacha@xxxxxxxxxx> 写道：

On 8 November 2017 at 17:16, Yao Guotao <yaoguo_tao@xxxxxxx> wrote:
Hi all,
I'm glad to add glusterfs community.

I have a glusterfs cluster:
Nodes: 4 
System: Centos7.1
Glusterfs: 3.8.9
Each Node:
CPU: 48 core
Mem: 128GB
Disk: 1*4T

There is one Distributed Replicated volume. There are ~160 k8s pods as clients connecting to glusterfs. But, the memory of  glusterfsd process is too high, gradually increase to 100G every node.
Then, I reboot the glusterfsd process. But the memory increase during approximate a week.
How can I debug the problem?

Hi,
Please take statedumps at intervals (a minimum of 2 at intervals of an hour) of a brick process for which you see the memory increasing and send them to us.  [1] describes how to take statedumps.

Regards,
Nithya

[1] http://docs.gluster.org/en/latest/Troubleshooting/statedump/ 

Thanks.

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users