|
Hello, We have found that the memory usage of glusterfsd is increasing when directories are created. Environment details can seen in the attached environment.txt file, and the translator graph can be seen in the attached file translator_graph.txt.
In summary we have a setup of two gluster bricks, using replicate mode, and one third gluster peer for quorum. We use glusterfs version 3.6.9. We build glusterfs directly from source code. The test script creates many directories on the /mnt/export volume from the storage client side. It creates directories of the path “/mnt/export/dir.XXXXX”, so all directories are under the same folder. In each directory a file of a random
size between 1024 and 11204 is created. The number of directories created varied from 10000 to 200000 for different runs. Before each run all directories were deleted. A graph of the memory usage can be seen here:
http://imgur.com/z8LOskQ In the X-axis the number of directories is seen. The number varies because before each run all directories are deleted. So first 80000 directories were created, then they were deleted and 160000 were created then deleted and 200000 were
created, then deleted and 40000 were created etc. Due to the amount of data the resolution of the X-axis is not fine enough to show the exact variation, but the general idea is that thousands of directories were created and deleted. The Y-axis shows the RSS memory usage of the glusterfsd process of this volume on the two storage nodes (SN-0 and SN-1), as measured by ‘pidstat –urdh –p <pid>’.
We can see that the memory usage is continually increasing. 1)
Is this increase normal expected? I can imagine that glusterfsd would allocated more memory if more files or directories are created, but is there an upper limit, or will it increase forever? 2)
Can we control the memory allocation by any configuration parameters? 3)
Is there a particular translator that controls this memory allocation? We did another test, where only files were created. In this case the memory usage reached an upper limit. Graph is available here:
http://imgur.com/zDtkbml So if only files are created there is an upper limit in memory allocation. Is the case of directory creation different and why? Or is this a memory leak? Has anyone else seen this behavior? Best regards, Klearchos |
# gluster --version glusterfs 3.6.9 built on Apr 14 2016 12:31:43 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. # uname -a Linux SN-0 4.1.20-pc64-distro.git-v2.11-sctpmh #1 SMP Thu Apr 14 11:59:31 UTC 2016 x86_64 GNU/Linux # gluster volume info export Volume Name: export Type: Replicate Volume ID: f5b5173d-f7d3-434c-a9da-fff6b617e21c Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: XXX.XXX.XXX.7:/mnt/bricks/export/brick Brick2: XXX.XXX.XXX.10:/mnt/bricks/export/brick Options Reconfigured: cluster.server-quorum-type: server cluster.consistent-metadata: on server.allow-insecure: on cluster.server-quorum-ratio: 51% # gluster volume status export Status of volume: export Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick XXX.XXX.XXX.7:/mnt/bricks/export/brick 49153 Y 11881 Brick XXX.XXX.XXX.10:/mnt/bricks/export/brick 49153 Y 25649 NFS Server on localhost N/A N N/A Self-heal Daemon on localhost N/A Y 7413 NFS Server on XXX.XXX.XXX.4 N/A N N/A Self-heal Daemon on XXX.XXX.XXX.4 N/A Y 16504 NFS Server on XXX.XXX.XXX.10 N/A N N/A Self-heal Daemon on XXX.XXX.XXX.10 N/A N N/A Task Status of Volume export ------------------------------------------------------------------------------ There are no active volume tasks # gluster volume list volume1 export volume2 volume3 volume4 volume5
============================ /mnt/export: Client graph ================== Final graph: +------------------------------------------------------------------------------+ 1: volume export-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 169.254.0.14 5: option remote-subvolume /mnt/bricks/export/brick 6: option transport-type socket 7: option send-gids true 8: end-volume 9: 10: volume export-client-1 11: type protocol/client 12: option ping-timeout 42 13: option remote-host 169.254.0.7 14: option remote-subvolume /mnt/bricks/export/brick 15: option transport-type socket 16: option send-gids true 17: end-volume 18: 19: volume export-replicate-0 20: type cluster/replicate 21: option consistent-metadata on 22: subvolumes export-client-0 export-client-1 23: end-volume 24: 25: volume export-dht 26: type cluster/distribute 27: subvolumes export-replicate-0 28: end-volume 29: 30: volume export-write-behind 31: type performance/write-behind 32: subvolumes export-dht 33: end-volume 34: 35: volume export-read-ahead 36: type performance/read-ahead 37: subvolumes export-write-behind 38: end-volume 39: 40: volume export-io-cache 41: type performance/io-cache 42: subvolumes export-read-ahead 43: end-volume 44: 45: volume export-quick-read 46: type performance/quick-read 47: subvolumes export-io-cache 48: end-volume 49: 50: volume export-open-behind 51: type performance/open-behind 52: subvolumes export-quick-read 53: end-volume 54: 55: volume export-md-cache 56: type performance/md-cache 57: subvolumes export-open-behind 58: end-volume 59: 60: volume export 61: type debug/io-stats 62: option latency-measurement off 63: option count-fop-hits off 64: subvolumes export-md-cache 65: end-volume 66: 67: volume meta-autoload 68: type meta 69: subvolumes export 70: end-volume 71: +------------------------------------------------------------------------------+ ============================ /mnt/export: Server graph ===================== Final graph: +------------------------------------------------------------------------------+ 1: volume export-posix 2: type storage/posix 3: option glusterd-uuid 1e0b6158-ff2b-4acc-8547-547c4754b32c 4: option directory /mnt/bricks/export/brick 5: option volume-id f5b5173d-f7d3-434c-a9da-fff6b617e21c 6: end-volume 7: 8: volume export-changelog 9: type features/changelog 10: option changelog-brick /mnt/bricks/export/brick 11: option changelog-dir /mnt/bricks/export/brick/.glusterfs/changelogs 12: option changelog-barrier-timeout 120 13: subvolumes export-posix 14: end-volume 15: 16: volume export-access-control 17: type features/access-control 18: subvolumes export-changelog 19: end-volume 20: 21: volume export-locks 22: type features/locks 23: subvolumes export-access-control 24: end-volume 25: 26: volume export-io-threads 27: type performance/io-threads 28: subvolumes export-locks 29: end-volume 30: 31: volume export-barrier 32: type features/barrier 33: option barrier disable 34: option barrier-timeout 120 34: option barrier-timeout 120 35: subvolumes export-io-threads 36: end-volume 37: 38: volume export-index 39: type features/index 40: option index-base /mnt/bricks/export/brick/.glusterfs/indices 41: subvolumes export-barrier 42: end-volume 43: 44: volume export-marker 45: type features/marker 46: option volume-uuid f5b5173d-f7d3-434c-a9da-fff6b617e21c 47: option timestamp-file /var/lib/glusterd/vols/export/marker.tstamp 48: option xtime off 49: option gsync-force-xtime off 50: option quota off 51: subvolumes export-index 52: end-volume 53: 54: volume export-quota 55: type features/quota 56: option volume-uuid export 57: option server-quota off 58: option timeout 0 59: option deem-statfs off 60: subvolumes export-marker 61: end-volume 62: 63: volume /mnt/bricks/export/brick 64: type debug/io-stats 65: option latency-measurement off 66: option count-fop-hits off 67: subvolumes export-quota 68: end-volume 69: 70: volume export-server 71: type protocol/server 72: option transport.socket.listen-port 49153 73: option rpc-auth.auth-glusterfs on 74: option rpc-auth.auth-unix on 75: option rpc-auth.auth-null on 76: option transport-type tcp 77: option auth.login./mnt/bricks/export/brick.allow aec4710a-db1a-42e6-aa4b-af6d80839ef2 78: option auth.login.aec4710a-db1a-42e6-aa4b-af6d80839ef2.password a76c8d5a-3055-48dd-9d6a-f50f371ccf6e 79: option auth.addr./mnt/bricks/export/brick.allow * 80: option rpc-auth-allow-insecure on 81: subvolumes /mnt/bricks/export/brick 82: end-volume 83: +------------------------------------------------------------------------------+
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
