A little late as it went out of my radar, apologies.
>From st02's glusterd log, it looks like profile commands were executed
very frequently.
[2016-06-02 10:35:02.232596] I [MSGID: 106494]
[glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume]
0-management: Received volume profile req for volume webroot
[2016-06-02 10:35:10.315758] I [MSGID: 106494]
[glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume]
0-management: Received volume profile req for volume webroot
[2016-06-02 10:35:10.487004] I [MSGID: 106494]
[glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume]
0-management: Received volume profile req for volume webroot
The thing which worries me here is the following log:
[2016-06-02 10:40:08.113106] W
[glusterd-locks.c:577:glusterd_mgmt_v3_lock]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(glusterd_op_sm+0x266)
[0x7fa5ad038106]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(+0x5aab5)
[0x7fa5ad02aab5]
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_lock+0x54d)
[0x7fa5ad0c097d] ) 0-management: Lock for webroot held by
86cc4dab-c13a-40b1-ac40-0f6a98c1b46c
[2016-06-02 10:40:08.113123] E [MSGID: 106119]
[glusterd-op-sm.c:3711:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for webroot
1758 [2016-06-02 10:40:08.113151] E [MSGID: 106376]
[glusterd-op-sm.c:7591:glusterd_op_sm] 0-management: handler returned: -1
This indicates that op-sm framework couldn't process the event and I
guess didn't free up the transaction object which got malloced. From
glusterd statedump it does look like there are so many txn objects which
are not freed up.
[mgmt/glusterd.management - usage-type gf_common_mt_txn_opinfo_obj_t
memusage]
size=3759056
*num_allocs=33563*
max_size=3759168
max_num_allocs=33564
total_allocs=167742
We are aware of some memory leak for gluster profile command but not at
this level and this needs a thorough investigation. Do you mind filing
up a bug for this and we pick that up for further analysis.
Thanks!
~Atin
On 06/02/2016 04:52 PM, moka111@xxxxxx wrote:
> Hi,
>
> here is the collected data requested by atinm in irc @ #gluster-dev.
> Additionally I can say, that all 5 nodes have 32GB of RAM installed.
>
> Regards
> moka111
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users
>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
