Dear All, We meet an issue with our storage infrastructure since I enabled quota service for our main storage volume. Indeed, since quota service is activated, when we try to write a new file on a defined quota path, we obtain this kind of message « Transport endpoint is not connected » -which disappears if we disable the quota service on the volume (or on the targeted subdirectory quota) I’m also noting no quota daemons seems to work on each bricks. Here is some information about my storage volumes: (i’ve highlighted the information which surprises me) [root@hades ~]# gluster volume status vol_home Status of volume: vol_home Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick ib-storage1:/export/brick_home/brick1 49164 Y 7373 Brick ib-storage2:/export/brick_home/brick1 49160 Y 6809 Brick ib-storage3:/export/brick_home/brick1 49152 Y 3436 Brick ib-storage4:/export/brick_home/brick1 49152 Y 3315 Brick ib-storage1:/export/brick_home/brick2 49166 Y 7380 Brick ib-storage2:/export/brick_home/brick2 49162 Y 6815 Brick ib-storage3:/export/brick_home/brick2 49154 Y 3440 Brick ib-storage4:/export/brick_home/brick2 49154 Y 3319 Self-heal Daemon on localhost N/A Y 22095 Quota Daemon on localhost N/A N N/A Self-heal Daemon on ib-storage3 N/A Y 16370 Quota Daemon on ib-storage3 N/A N N/A Self-heal Daemon on 10.0.4.1 N/A Y 14686 Quota Daemon on 10.0.4.1 N/A N N/A Self-heal Daemon on ib-storage4 N/A Y 16172 Quota Daemon on ib-storage4 N/A N N/A Task Status of Volume vol_home ------------------------------------------------------------------------------ There are no active volume tasks [root@hades ~]# gluster volume status vol_home detail Status of volume: vol_home ------------------------------------------------------------------------------ Brick : Brick ib-storage1:/export/brick_home/brick1 Port : 49164 Online : Y Pid : 7373 File System : xfs Device : /dev/mapper/storage1--block1-st1--blk1--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 6.9TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845133649 ------------------------------------------------------------------------------ Brick : Brick ib-storage2:/export/brick_home/brick1 Port : 49160 Online : Y Pid : 6809 File System : xfs Device : /dev/mapper/storage2--block1-st2--blk1--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 6.9TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845133649 ------------------------------------------------------------------------------ Brick : Brick ib-storage3:/export/brick_home/brick1 Port : 49152 Online : Y Pid : 3436 File System : xfs Device : /dev/mapper/storage3--block1-st3--blk1--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 7.4TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845131362 ------------------------------------------------------------------------------ Brick : Brick ib-storage4:/export/brick_home/brick1 Port : 49152 Online : Y Pid : 3315 File System : xfs Device : /dev/mapper/storage4--block1-st4--blk1--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 7.4TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845131363 ------------------------------------------------------------------------------ Brick : Brick ib-storage1:/export/brick_home/brick2 Port : 49166 Online : Y Pid : 7380 File System : xfs Device : /dev/mapper/storage1--block2-st1--blk2--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 6.8TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845128559 ------------------------------------------------------------------------------ Brick : Brick ib-storage2:/export/brick_home/brick2 Port : 49162 Online : Y Pid : 6815 File System : xfs Device : /dev/mapper/storage2--block2-st2--blk2--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 6.8TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845128559 ------------------------------------------------------------------------------ Brick : Brick ib-storage3:/export/brick_home/brick2 Port : 49154 Online : Y Pid : 3440 File System : xfs Device : /dev/mapper/storage3--block2-st3--blk2--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 7.0TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845124761 ------------------------------------------------------------------------------ Brick : Brick ib-storage4:/export/brick_home/brick2 Port : 49154 Online : Y Pid : 3319 File System : xfs Device : /dev/mapper/storage4--block2-st4--blk2--home Mount Options : rw,noatime,nodiratime,attr2,quota Inode Size : 256 Disk Space Free : 7.0TB Total Disk Space : 17.9TB Inode Count : 3853515968 Free Inodes : 3845124761 [root@hades ~]# gluster volume info vol_home Volume Name: vol_home Type: Distributed-Replicate Volume ID: f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp,rdma Bricks: Brick1: ib-storage1:/export/brick_home/brick1 Brick2: ib-storage2:/export/brick_home/brick1 Brick3: ib-storage3:/export/brick_home/brick1 Brick4: ib-storage4:/export/brick_home/brick1 Brick5: ib-storage1:/export/brick_home/brick2 Brick6: ib-storage2:/export/brick_home/brick2 Brick7: ib-storage3:/export/brick_home/brick2 Brick8: ib-storage4:/export/brick_home/brick2 Options Reconfigured: diagnostics.brick-log-level: CRITICAL auth.allow: localhost,127.0.0.1,10.* nfs.disable: on performance.cache-size: 64MB performance.write-behind-window-size: 1MB performance.quick-read: on performance.io-cache: on performance.io-thread-count: 64 features.quota: on As you can read below, the CLI doesn’t show me the quota list (even after waiting a couple of hours) but i can get quota information specifying the quota path. [root@hades ~]# gluster volume quota vol_home list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- ^C [root@hades ~]# gluster volume quota vol_home list /admin_team Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /admin_team 1.0TB 80% 3.6GB 1020.4GB And additionally, i note a quota-crawl.log file is growing bigger… For information: - all storage node are running CentOS 6.5 - previously on the same servers we were running a 3.3 GlusterFS version but, after having removing the old versions of GlusterFS packages and rebuilding all the bricks physically (RAID60->RAID6 and multiplying per 2 my storage nodes count and reimporting all my data inside this new volume), I installed 3.5.2 GlusterFS version. —Of course all storage node have been restarted several times since the upgrade -> but i note a troubling thing in gluster log file : [root@hades ~]# gluster --version glusterfs 3.5.2 built on Jul 31 2014 18:47:54 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@hades ~]# cat /var/log/glusterfs/home.log|grep "version numbers are not same"|tail -1l [2014-09-15 10:35:35.516925] I [client-handshake.c:1474:client_setvolume_cbk] 3-vol_home-client-7: Server and Client lk-version numbers are not same, reopening the fds [root@hades ~]# cat /var/log/glusterfs/home.log|grep "GlusterFS 3.3"|tail -1l [2014-09-15 10:35:35.516082] I [client-handshake.c:1677:select_server_supported_programs] 3-vol_home-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) [root@hades ~]# rpm -qa gluster* glusterfs-fuse-3.5.2-1.el6.x86_64 glusterfs-rdma-3.5.2-1.el6.x86_64 glusterfs-3.5.2-1.el6.x86_64 glusterfs-server-3.5.2-1.el6.x86_64 glusterfs-libs-3.5.2-1.el6.x86_64 glusterfs-cli-3.5.2-1.el6.x86_64 glusterfs-api-3.5.2-1.el6.x86_64 Can someone help me to fix the problem? Thanks in advance and have a nice day, Geoffrey PS: Don’t hesitate to tell me if you see something wrong (or better to do) in my volume settings. ------------------------------------------------------ Geoffrey Letessier Responsable informatique UPR 9080 - CNRS - Laboratoire de Biochimie Théorique Institut de Biologie Physico-Chimique 13, rue Pierre et Marie Curie - 75005 Paris |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
