Memory usage of glusterfsd increases when directories are created

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

 

We have found that the memory usage of glusterfsd is increasing when directories are created. Environment details can seen in the attached environment.txt file, and the translator graph can be seen in the attached file translator_graph.txt. In summary we have a setup of two gluster bricks, using replicate mode, and one third gluster peer for quorum. We use glusterfs version 3.6.9. We build glusterfs directly from source code.

 

The test script creates many directories on the /mnt/export volume from the storage client side. It creates directories of the path “/mnt/export/dir.XXXXX”, so all directories are under the same folder. In each directory a file of a random size between 1024 and 11204 is created.

The number of directories created varied from 10000 to 200000 for different runs. Before each run all directories were deleted.

 

A graph of the memory usage can be seen here: http://imgur.com/z8LOskQ

 

In the X-axis the number of directories is seen. The number varies because before each run all directories are deleted. So first 80000 directories were created, then they were deleted and 160000 were created then deleted and 200000 were created, then deleted and 40000 were created etc. Due to the amount of data the resolution of the X-axis is not fine enough to show the exact variation, but the general idea is that thousands of directories were created and deleted.

 

The Y-axis shows the RSS memory usage of the glusterfsd process of this volume on the two storage nodes (SN-0 and SN-1), as measured by ‘pidstat –urdh –p <pid>’.

We can see that the memory usage is continually increasing.

 

1)     Is this increase normal expected? I can imagine that glusterfsd would allocated more memory if more files or directories are created, but is there an upper limit, or will it increase forever?

2)     Can we control the memory allocation by any configuration parameters?

3)     Is there a particular translator that controls this memory allocation?

 

We did another test, where only files were created. In this case the memory usage reached an upper limit. Graph is available here: http://imgur.com/zDtkbml

 

So if only files are created there is an upper limit in memory allocation. Is the case of directory creation different and why? Or is this a memory leak? Has anyone else seen this behavior?

 

Best regards,

Klearchos

# gluster --version
glusterfs 3.6.9 built on Apr 14 2016 12:31:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

# uname -a
Linux SN-0 4.1.20-pc64-distro.git-v2.11-sctpmh #1 SMP Thu Apr 14 11:59:31 UTC 2016 x86_64 GNU/Linux

# gluster volume info export
 
Volume Name: export
Type: Replicate
Volume ID: f5b5173d-f7d3-434c-a9da-fff6b617e21c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: XXX.XXX.XXX.7:/mnt/bricks/export/brick
Brick2: XXX.XXX.XXX.10:/mnt/bricks/export/brick
Options Reconfigured:
cluster.server-quorum-type: server
cluster.consistent-metadata: on
server.allow-insecure: on
cluster.server-quorum-ratio: 51%

# gluster volume status export
Status of volume: export
Gluster process                                 Port    Online   Pid
------------------------------------------------------------------------------
Brick XXX.XXX.XXX.7:/mnt/bricks/export/brick    49153   Y        11881
Brick XXX.XXX.XXX.10:/mnt/bricks/export/brick   49153   Y        25649
NFS Server on localhost                         N/A	    N        N/A
Self-heal Daemon on localhost                   N/A     Y        7413
NFS Server on XXX.XXX.XXX.4                     N/A     N        N/A
Self-heal Daemon on XXX.XXX.XXX.4               N/A     Y        16504
NFS Server on XXX.XXX.XXX.10                    N/A     N        N/A
Self-heal Daemon on XXX.XXX.XXX.10              N/A     N        N/A
 
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks

# gluster volume list
volume1
export
volume2
volume3
volume4
volume5
============================ /mnt/export: Client graph ==================

Final graph:
+------------------------------------------------------------------------------+
  1: volume export-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host 169.254.0.14
  5:     option remote-subvolume /mnt/bricks/export/brick
  6:     option transport-type socket
  7:     option send-gids true
  8: end-volume
  9:
 10: volume export-client-1
 11:     type protocol/client
 12:     option ping-timeout 42
 13:     option remote-host 169.254.0.7
 14:     option remote-subvolume /mnt/bricks/export/brick
 15:     option transport-type socket
 16:     option send-gids true
 17: end-volume
 18:
 19: volume export-replicate-0
 20:     type cluster/replicate
 21:     option consistent-metadata on
 22:     subvolumes export-client-0 export-client-1
 23: end-volume
 24:
 25: volume export-dht
 26:     type cluster/distribute
 27:     subvolumes export-replicate-0
 28: end-volume
 29:
 30: volume export-write-behind
 31:     type performance/write-behind
 32:     subvolumes export-dht
 33: end-volume
 34:
 35: volume export-read-ahead
 36:     type performance/read-ahead
 37:     subvolumes export-write-behind
 38: end-volume
 39:
 40: volume export-io-cache
 41:     type performance/io-cache
 42:     subvolumes export-read-ahead
 43: end-volume
 44:
 45: volume export-quick-read
 46:     type performance/quick-read
 47:     subvolumes export-io-cache
 48: end-volume
 49:
 50: volume export-open-behind
 51:     type performance/open-behind
 52:     subvolumes export-quick-read
 53: end-volume
 54:
 55: volume export-md-cache
 56:     type performance/md-cache
 57:     subvolumes export-open-behind
 58: end-volume
 59:
 60: volume export
 61:     type debug/io-stats
 62:     option latency-measurement off
 63:     option count-fop-hits off
 64:     subvolumes export-md-cache
 65: end-volume
 66:
 67: volume meta-autoload
 68:     type meta
 69:     subvolumes export
 70: end-volume
 71:
+------------------------------------------------------------------------------+



============================ /mnt/export: Server graph =====================
Final graph:
+------------------------------------------------------------------------------+
  1: volume export-posix
  2:     type storage/posix
  3:     option glusterd-uuid 1e0b6158-ff2b-4acc-8547-547c4754b32c
  4:     option directory /mnt/bricks/export/brick
  5:     option volume-id f5b5173d-f7d3-434c-a9da-fff6b617e21c
  6: end-volume
  7:
  8: volume export-changelog
  9:     type features/changelog
 10:     option changelog-brick /mnt/bricks/export/brick
 11:     option changelog-dir /mnt/bricks/export/brick/.glusterfs/changelogs
 12:     option changelog-barrier-timeout 120
 13:     subvolumes export-posix
 14: end-volume
 15:
 16: volume export-access-control
 17:     type features/access-control
 18:     subvolumes export-changelog
 19: end-volume
 20:
 21: volume export-locks
 22:     type features/locks
 23:     subvolumes export-access-control
 24: end-volume
 25:
 26: volume export-io-threads
 27:     type performance/io-threads
 28:     subvolumes export-locks
 29: end-volume
 30:
 31: volume export-barrier
 32:     type features/barrier
 33:     option barrier disable
 34:     option barrier-timeout 120
 34:     option barrier-timeout 120
 35:     subvolumes export-io-threads
 36: end-volume
 37:
 38: volume export-index
 39:     type features/index
 40:     option index-base /mnt/bricks/export/brick/.glusterfs/indices
 41:     subvolumes export-barrier
 42: end-volume
 43:
 44: volume export-marker
 45:     type features/marker
 46:     option volume-uuid f5b5173d-f7d3-434c-a9da-fff6b617e21c
 47:     option timestamp-file /var/lib/glusterd/vols/export/marker.tstamp
 48:     option xtime off
 49:     option gsync-force-xtime off
 50:     option quota off
 51:     subvolumes export-index
 52: end-volume
 53:
 54: volume export-quota
 55:     type features/quota
 56:     option volume-uuid export
 57:     option server-quota off
 58:     option timeout 0
 59:     option deem-statfs off
 60:     subvolumes export-marker
 61: end-volume
 62:
 63: volume /mnt/bricks/export/brick
 64:     type debug/io-stats
 65:     option latency-measurement off
 66:     option count-fop-hits off
 67:     subvolumes export-quota
 68: end-volume
 69:
 70: volume export-server
 71:     type protocol/server
 72:     option transport.socket.listen-port 49153
 73:     option rpc-auth.auth-glusterfs on
 74:     option rpc-auth.auth-unix on
 75:     option rpc-auth.auth-null on
 76:     option transport-type tcp
 77:     option auth.login./mnt/bricks/export/brick.allow aec4710a-db1a-42e6-aa4b-af6d80839ef2
 78:     option auth.login.aec4710a-db1a-42e6-aa4b-af6d80839ef2.password a76c8d5a-3055-48dd-9d6a-f50f371ccf6e
 79:     option auth.addr./mnt/bricks/export/brick.allow *
 80:     option rpc-auth-allow-insecure on
 81:     subvolumes /mnt/bricks/export/brick
 82: end-volume
 83:
+------------------------------------------------------------------------------+


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux