Hi, Can you provide the statedump of the process, it can be obtained as follows: $ gluster --print-statedumpdir #create this directory if it doesn't exist. $ kill -USR1 <pid-of-glusterfs-process> #generates state dump. Also, xporting Gluster via Samba-VFS-plugin method is preferred over Fuse mount export. For more details refer to: http://lalatendumohanty.wordpress.com/2014/02/11/using-glusterfs-with-samba-and-samba-vfs-plugin-for-glusterfs-on-fedora-20/ Regards, Poornima ----- Original Message ----- From: "Tamas Papp" <tompos@xxxxxxxxxxxxx> To: Gluster-users@xxxxxxxxxxx Sent: Thursday, July 31, 2014 12:07:36 AM Subject: high memory usage of mount hi All, I have a small cluster. It's exported to windows clients via smb though local mount. 2464 root 20 0 16.8g 11g 1292 S 0 71.3 33:09.14 /usr/sbin/glusterfs --direct-io-mode=disable --volfile-id=/w-vol --volfile-server=gl0 /W/Projects 2212 root 20 0 1491m 25m 844 S 3 0.2 18:37.37 /usr/sbin/glusterfsd -s gl0 --volfile-id w-vol.gl0.mnt-brick1-export -p /var/lib/glusterd/vols/w-v total used free shared buffers cached Mem: 16422548 15956152 466396 0 5400 751024 -/+ buffers/cache: 15199728 1222820 Swap: 5859324 5756832 102492 The mount process log (W-Projects.log) is full off these errors: /3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobref [2014-07-30 17:04:24.082004] E [iobuf.c:733:iobuf_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x182) [0x7fcb6d161022] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x16b) [0x7fcb6d377d0b] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobuf [2014-07-30 17:04:24.082036] E [iobuf.c:828:iobref_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x182) [0x7fcb6d161022] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x16b) [0x7fcb6d377d0b] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobref [2014-07-30 17:04:24.085388] E [iobuf.c:733:iobuf_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x182) [0x7fcb6d161022] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x16b) [0x7fcb6d377d0b] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobuf [2014-07-30 17:04:24.085419] E [iobuf.c:828:iobref_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x182) [0x7fcb6d161022] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x16b) [0x7fcb6d377d0b] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobref [2014-07-30 17:04:24.246769] E [iobuf.c:733:iobuf_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/debug/io-stats.so(io_stats_readv+0x182) [0x7fcb6d161022] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/md-cache.so(mdc_readv+0x16b) [0x7fcb6d377d0b] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.4.4/xlator/performance/quick-read.so(qr_readv+0x62) [0x7fcb6d5852e2]))) 0-iobuf: invalid argument: iobuf rebalance.log: [2014-07-06 22:28:56.460558] I [dht-common.c:2614:dht_setxattr] 0-w-vol-dht: fixing the layout of /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp [2014-07-06 22:28:56.463787] I [dht-common.c:2614:dht_setxattr] 0-w-vol-dht: fixing the layout of /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001 [2014-07-06 22:28:56.466934] W [dht-common.c:419:dht_lookup_dir_cbk] 0-w-vol-dht: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800: gfid different on w-vol-client-1 [2014-07-06 22:28:56.467127] I [dht-common.c:2614:dht_setxattr] 0-w-vol-dht: fixing the layout of /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 [2014-07-06 22:28:56.467553] W [client-rpc-fops.c:1044:client3_3_setxattr_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory [2014-07-06 22:28:56.468787] W [client-rpc-fops.c:1994:client3_3_setattr_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory [2014-07-06 22:28:56.468892] E [dht-selfheal.c:1015:dht_dir_attr_heal] 0-dht: Failed to set uid/gid on /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 on w-vol-client-1 subvol (No such file or directory) [2014-07-06 22:28:56.468906] W [client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory. Path: /_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 (00000000-0000-0000-0000-000000000000) [2014-07-06 22:28:56.470407] W [client-rpc-fops.c:2329:client3_3_readdirp_cbk] 0-w-vol-client-1: remote operation failed: No such file or directory [2014-07-06 22:28:56.518470] I [dht-rebalance.c:1729:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1350.00 secs [2014-07-06 22:28:56.518494] I [dht-rebalance.c:1732:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0 [2014-07-06 22:28:56.527984] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f71868543fd] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f7186b27e9a] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x7f7187611e75]))) 0-: received signum (15), shutting down /_torolni/ directory content is not deletable. Volume Name: w-vol Type: Distribute Volume ID: 89e31546-cc2e-4a27-a448-17befda04726 Status: Started Number of Bricks: 5 Transport-type: tcp Bricks: Brick1: gl0:/mnt/brick1/export Brick2: gl1:/mnt/brick1/export Brick3: gl2:/mnt/brick1/export Brick4: gl3:/mnt/brick1/export Brick5: gl4:/mnt/brick1/export Options Reconfigured: nfs.mount-udp: on nfs.addr-namelookup: off nfs.ports-insecure: on nfs.port: 2049 cluster.stripe-coalesce: on nfs.disable: off performance.flush-behind: on performance.io-thread-count: 64 performance.quick-read: on performance.stat-prefetch: on performance.io-cache: on performance.write-behind: on performance.read-ahead: on performance.write-behind-window-size: 4MB performance.cache-refresh-timeout: 1 performance.cache-size: 4GB network.frame-timeout: 60 performance.cache-max-file-size: 1GB ii glusterfs-client 3.4.4-ubuntu1~precise1 clustered file-system (client package) hi glusterfs-common 3.4.4-ubuntu1~precise1 GlusterFS common libraries and translator modules ii glusterfs-server 3.4.4-ubuntu1~precise1 clustered file-system (server package) No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 12.04.4 LTS Release: 12.04 Codename: precise Linux gl0 3.11.0-26-generic #45~precise1-Ubuntu SMP Tue Jul 15 04:02:35 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Do you have any idea, what's going on? Thanks, tamas _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
