Re: glusterfs brick server use too high memory

["Yao Guotao" <yaoguo_tao@xxxxxxx>]

Thank you very much for your reply.

I think I found the key to the problem. But, I'd like you to confirm whether this is the issue.

My gluster version is 3.8.9, And I found there is a bug in it, and fixed in 3.8.11:

https://bugzilla.redhat.com/show_bug.cgi?id=1431592

A large amount of memory allocated for agf_common_mt_strdup. Everythig else seem to be all right.

My cluster:

Yesterday Afternoon：（I'm sorry I forgot the specific time.）

[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=1941483443
num_allocs=617382139
max_size=1941483443
max_num_allocs=617382139
total_allocs=661873332

TIme: 2017.11.9 17:15 (Today)

[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=792538295
num_allocs=752904534
max_size=792538295
max_num_allocs=752904534
total_allocs=800889589

The statedump files are in the attachment.

Thanks again.

在 2017-11-09 17:22:03，"Nithya Balachandran" <nbalacha@xxxxxxxxxx> 写道：

On 8 November 2017 at 17:16, Yao Guotao <yaoguo_tao@xxxxxxx> wrote:

Hi all,

I'm glad to add glusterfs community.

I have a glusterfs cluster:

Nodes: 4

System: Centos7.1

Glusterfs: 3.8.9

Each Node:

CPU: 48 core

Mem: 128GB

Disk: 1*4T

There is one Distributed Replicated volume. There are ~160 k8s pods as clients connecting to glusterfs. But, the memory of  glusterfsd process is too high, gradually increase to 100G every node.

Then, I reboot the glusterfsd process. But the memory increase during approximate a week.

How can I debug the problem?

Hi,

Please take statedumps at intervals (a minimum of 2 at intervals of an hour) of a brick process for which you see the memory increasing and send them to us.  [1] describes how to take statedumps.

Regards,

Nithya

[1] http://docs.gluster.org/en/latest/Troubleshooting/statedump/ 

 

Thanks.

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

 

Attachment: home-www-log.152061.dump.20171108
Description: Binary data

Attachment: home-www-log.152061.dump.20171109
Description: Binary data

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users