Re: Memory leak in GlusterFS FUSE client

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



What units Cache_Size is measured in? Bytes?

25.12.2015 16:58, Soumya Koduri написав:
On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:

===
root      5416 34.2 78.5 2047176 1480552 ?     Ssl  12:02 117:54
/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
/etc/ganesha/ganesha.conf -N NIV_EVENT
===

1.4G is too much for simple stat() :(.

Ideas?
nfs-ganesha also has cache layer which can scale to millions of
entries depending on the number of files/directories being looked
upon. However there are parameters to tune it. So either try stat with
few entries or add below block in nfs-ganesha.conf file, set low
limits and check the difference. That may help us narrow down how much
memory actually consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
	Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
	Entries_HWMark(uint32, range 1 to UINT32_MAX, default 100000); #Max
no. of entries in the cache.
}

Thanks,
Soumya


24.12.2015 16:32, Oleksandr Natalenko написав:
Still actual issue for 3.7.6. Any suggestions?

24.09.2015 10:14, Oleksandr Natalenko написав:
In our GlusterFS deployment we've encountered something like memory
leak in GlusterFS FUSE client.

We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
maildir format). Here is inode stats for both bricks and mountpoint:

===
Brick 1 (Server 1):

Filesystem Inodes IUsed
     IFree IUse% Mounted on
/dev/mapper/vg_vd1_misc-lv08_mail 578768144 10954918
 567813226    2% /bricks/r6sdLV08_vd1_mail

Brick 2 (Server 2):

Filesystem Inodes IUsed
     IFree IUse% Mounted on
/dev/mapper/vg_vd0_misc-lv07_mail 578767984 10954913
 567813071    2% /bricks/r6sdLV07_vd0_mail

Mountpoint (Server 3):

Filesystem                              Inodes    IUsed      IFree
IUse% Mounted on
glusterfs.xxx:mail                   578767760 10954915  567812845
2% /var/spool/mail/virtual
===

glusterfs.xxx domain has two A records for both Server 1 and Server 2.

Here is volume info:

===
Volume Name: mail
Type: Replicate
Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Options Reconfigured:
nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
features.cache-invalidation-timeout: 10
performance.stat-prefetch: off
performance.quick-read: on
performance.read-ahead: off
performance.flush-behind: on
performance.write-behind: on
performance.io-thread-count: 4
performance.cache-max-file-size: 1048576
performance.cache-size: 67108864
performance.readdir-ahead: off
===

Soon enough after mounting and exim/dovecot start, glusterfs client
process begins to consume huge amount of RAM:

===
user@server3 ~$ ps aux | grep glusterfs | grep mail
root     28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
/usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
--volfile-server=glusterfs.xxx --volfile-id=mail
/var/spool/mail/virtual
===

That is, ~15 GiB of RAM.

Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
GiB of RAM, and soon after starting mail daemons got OOM killer for
glusterfs client process.

Mounting same share via NFS works just fine. Also, we have much less
iowait and loadavg on client side with NFS.

Also, we've tried to change IO threads count and cache size in order
to limit memory usage with no luck. As you can see, total cache size
is 4×64==256 MiB (compare to 15 GiB).

Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
help as well.

Here are volume memory stats:

===
Memory status for volume : mail
----------------------------------------------
Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
Mallinfo
--------
Arena    : 36859904
Ordblks  : 10357
Smblks   : 519
Hblks    : 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 53440
Uordblks : 18604144
Fordblks : 18255760
Keepcost : 114112

Mempool Stats
-------------
Name                            HotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
----                            -------- --------- ------------
---------- -------- -------- ------------
mail-server:fd_t                       0      1024          108
30773120      137        0            0
mail-server:dentry_t               16110       274           84
235676148    16384  1106499         1152
mail-server:inode_t                16363        21          156
237216876    16384  1876651         1169
mail-trash:fd_t                        0      1024          108
  0        0        0            0
mail-trash:dentry_t                    0     32768           84
  0        0        0            0
mail-trash:inode_t                     4     32764          156
  4        4        0            0
mail-trash:trash_local_t               0        64         8628
  0        0        0            0
mail-changetimerecorder:gf_ctr_local_t         0        64
16540          0        0        0            0
mail-changelog:rpcsvc_request_t         0         8         2828
   0        0        0            0
mail-changelog:changelog_local_t         0        64          116
    0        0        0            0
mail-bitrot-stub:br_stub_local_t         0       512           84
79204        4        0            0
mail-locks:pl_local_t                  0        32          148
6812757        4        0            0
mail-upcall:upcall_local_t             0       512          108
  0        0        0            0
mail-marker:marker_local_t             0       128          332
64980        3        0            0
mail-quota:quota_local_t               0        64          476
  0        0        0            0
mail-server:rpcsvc_request_t           0       512         2828
45462533       34        0            0
glusterfs:struct saved_frame           0         8          124
  2        2        0            0
glusterfs:struct rpc_req               0         8          588
  2        2        0            0
glusterfs:rpcsvc_request_t             1         7         2828
  2        1        0            0
glusterfs:log_buf_t                    5       251          140
3452        6        0            0
glusterfs:data_t                     242     16141           52
480115498      664        0            0
glusterfs:data_pair_t                230     16153           68
179483528      275        0            0
glusterfs:dict_t                      23      4073          140
303751675      627        0            0
glusterfs:call_stub_t                  0      1024         3764
45290655       34        0            0
glusterfs:call_stack_t                 1      1023         1708
43598469       34        0            0
glusterfs:call_frame_t                 1      4095          172
336219655      184        0            0
----------------------------------------------
Brick : server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
Mallinfo
--------
Arena    : 38174720
Ordblks  : 9041
Smblks   : 507
Hblks    : 21
Hblkhd   : 30515200
Usmblks  : 0
Fsmblks  : 51712
Uordblks : 19415008
Fordblks : 18759712
Keepcost : 114848

Mempool Stats
-------------
Name                            HotCount ColdCount PaddedSizeof
AllocCount MaxAlloc   Misses Max-StdAlloc
----                            -------- --------- ------------
---------- -------- -------- ------------
mail-server:fd_t                       0      1024          108
2373075      133        0            0
mail-server:dentry_t               14114      2270           84
3513654    16384     2300          267
mail-server:inode_t                16374        10          156
6766642    16384   194635         1279
mail-trash:fd_t                        0      1024          108
  0        0        0            0
mail-trash:dentry_t                    0     32768           84
  0        0        0            0
mail-trash:inode_t                     4     32764          156
  4        4        0            0
mail-trash:trash_local_t               0        64         8628
  0        0        0            0
mail-changetimerecorder:gf_ctr_local_t         0        64
16540          0        0        0            0
mail-changelog:rpcsvc_request_t         0         8         2828
   0        0        0            0
mail-changelog:changelog_local_t         0        64          116
    0        0        0            0
mail-bitrot-stub:br_stub_local_t         0       512           84
71354        4        0            0
mail-locks:pl_local_t                  0        32          148
8135032        4        0            0
mail-upcall:upcall_local_t             0       512          108
  0        0        0            0
mail-marker:marker_local_t             0       128          332
65005        3        0            0
mail-quota:quota_local_t               0        64          476
  0        0        0            0
mail-server:rpcsvc_request_t           0       512         2828
12882393       30        0            0
glusterfs:struct saved_frame           0         8          124
  2        2        0            0
glusterfs:struct rpc_req               0         8          588
  2        2        0            0
glusterfs:rpcsvc_request_t             1         7         2828
  2        1        0            0
glusterfs:log_buf_t                    5       251          140
3443        6        0            0
glusterfs:data_t                     242     16141           52
138743429      290        0            0
glusterfs:data_pair_t                230     16153           68
126649864      270        0            0
glusterfs:dict_t                      23      4073          140
20356289       63        0            0
glusterfs:call_stub_t                  0      1024         3764
13678560       31        0            0
glusterfs:call_stack_t                 1      1023         1708
11011561       30        0            0
glusterfs:call_frame_t                 1      4095          172
125764190      193        0            0
----------------------------------------------
===

So, my questions are:

1) what one should do to limit GlusterFS FUSE client memory usage?
2) what one should do to prevent client high loadavg because of high
iowait because of multiple concurrent volume users?

Server/client OS is CentOS 7.1, GlusterFS server version is 3.7.3,
GlusterFS client version is 3.7.4.

Any additional info needed?
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux