Hi,
Apparently, version 3.6.9 is suffering from a SERIOUS memory
leak as illustrated in the following logs:
2016-04-26T11:54:27.971564+00:00 tsunami1 kernel:
[698635.210069] glusterfsd invoked oom-killer: gfp_mask=0x201da,
order=0, oom_score_adj=0
2016-04-26T11:54:27.974133+00:00 tsunami1 kernel:
[698635.210076] Pid: 28111, comm: glusterfsd Tainted: G W O
3.7.10-1.1-desktop #1
2016-04-26T11:54:27.974136+00:00 tsunami1 kernel:
[698635.210077] Call Trace:
2016-04-26T11:54:27.974137+00:00 tsunami1 kernel:
[698635.210090] [<ffffffff81004818>] dump_trace+0x88/0x300
2016-04-26T11:54:27.974137+00:00 tsunami1 kernel:
[698635.210096] [<ffffffff8158b033>] dump_stack+0x69/0x6f
2016-04-26T11:54:27.974138+00:00 tsunami1 kernel:
[698635.210101] [<ffffffff8158db39>]
dump_header+0x70/0x200
2016-04-26T11:54:27.974139+00:00 tsunami1 kernel:
[698635.210105] [<ffffffff81112ad4>]
oom_kill_process+0x244/0x390
2016-04-26T11:54:28.113125+00:00 tsunami1 kernel:
[698635.210111] [<ffffffff81113211>]
out_of_memory+0x451/0x490
2016-04-26T11:54:28.113142+00:00 tsunami1 kernel:
[698635.210116] [<ffffffff81118afe>]
__alloc_pages_nodemask+0x8ae/0x9f0
2016-04-26T11:54:28.113143+00:00 tsunami1 kernel:
[698635.210122] [<ffffffff81152fb7>]
alloc_pages_current+0xb7/0x130
2016-04-26T11:54:28.113144+00:00 tsunami1 kernel:
[698635.210127] [<ffffffff81111673>]
filemap_fault+0x283/0x440
2016-04-26T11:54:28.113144+00:00 tsunami1 kernel:
[698635.210131] [<ffffffff811345ee>] __do_fault+0x6e/0x560
2016-04-26T11:54:28.113145+00:00 tsunami1 kernel:
[698635.210136] [<ffffffff81137cf7>]
handle_pte_fault+0x97/0x490
2016-04-26T11:54:28.113145+00:00 tsunami1 kernel:
[698635.210141] [<ffffffff8159af8b>]
__do_page_fault+0x16b/0x4c0
2016-04-26T11:54:28.113562+00:00 tsunami1 kernel:
[698635.210145] [<ffffffff815982f8>] page_fault+0x28/0x30
2016-04-26T11:54:28.113565+00:00 tsunami1 kernel:
[698635.210158] [<00007fa9d8a8292b>] 0x7fa9d8a8292a
2016-04-26T11:54:28.120811+00:00 tsunami1 kernel:
[698635.226243] Out of memory: Kill process 17144 (glusterfsd)
score 694 or sacrifice child
2016-04-26T11:54:28.120811+00:00 tsunami1 kernel:
[698635.226251] Killed process 17144 (glusterfsd)
total-vm:8956384kB, anon-rss:6670900kB, file-rss:0kB
It makes this version completely useless in production. Bricks
servers have 8 GB of RAM (but will be upgraded to 16 GB).
gluster volume info <VOLUME> returns:
Volume Name: home
Type: Distributed-Replicate
Volume ID: 501741ed-4146-4022-af0b-41f5b1297766
Status: Started
Number of Bricks: 14 x 2 = 28
Transport-type: tcp
Bricks:
Brick1: tsunami1:/data/glusterfs/home/brick1
Brick2: tsunami2:/data/glusterfs/home/brick1
Brick3: tsunami1:/data/glusterfs/home/brick2
Brick4: tsunami2:/data/glusterfs/home/brick2
Brick5: tsunami1:/data/glusterfs/home/brick3
Brick6: tsunami2:/data/glusterfs/home/brick3
Brick7: tsunami1:/data/glusterfs/home/brick4
Brick8: tsunami2:/data/glusterfs/home/brick4
Brick9: tsunami3:/data/glusterfs/home/brick1
Brick10: tsunami4:/data/glusterfs/home/brick1
Brick11: tsunami3:/data/glusterfs/home/brick2
Brick12: tsunami4:/data/glusterfs/home/brick2
Brick13: tsunami3:/data/glusterfs/home/brick3
Brick14: tsunami4:/data/glusterfs/home/brick3
Brick15: tsunami3:/data/glusterfs/home/brick4
Brick16: tsunami4:/data/glusterfs/home/brick4
Brick17: tsunami5:/data/glusterfs/home/brick1
Brick18: tsunami6:/data/glusterfs/home/brick1
Brick19: tsunami5:/data/glusterfs/home/brick2
Brick20: tsunami6:/data/glusterfs/home/brick2
Brick21: tsunami5:/data/glusterfs/home/brick3
Brick22: tsunami6:/data/glusterfs/home/brick3
Brick23: tsunami5:/data/glusterfs/home/brick4
Brick24: tsunami6:/data/glusterfs/home/brick4
Brick25: tsunami7:/data/glusterfs/home/brick1
Brick26: tsunami8:/data/glusterfs/home/brick1
Brick27: tsunami7:/data/glusterfs/home/brick2
Brick28: tsunami8:/data/glusterfs/home/brick2
Options Reconfigured:
nfs.export-dir: /gerb-reproc/Archive
nfs.volume-access: read-only
cluster.ensure-durability: on
features.quota: on
performance.cache-size: 512MB
performance.io-thread-count: 32
performance.flush-behind: off
performance.write-behind-window-size: 4MB
performance.write-behind: off
nfs.disable: off
cluster.read-hash-mode: 2
diagnostics.brick-log-level: CRITICAL
cluster.lookup-unhashed: on
server.allow-insecure: on
auth.allow: localhost, <COUPLE OF IP ADDRESSES>
cluster.readdir-optimize: on
performance.readdir-ahead: on
nfs.export-volumes: off
Are you aware if this issue ?
Thanks,
A.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
