Hi, Can you upload the logs ? Susant ----- Original Message ----- From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx> To: "SINCOCK John" <J.Sincock@xxxxxxxxx>, gluster-users@xxxxxxxxxxx Cc: "Susant Palai" <spalai@xxxxxxxxxx> Sent: Wednesday, 18 June, 2014 7:48:19 AM Subject: Re: Unable to delete files but getfattr shows file is part of glusterfs I think Susant knows about this issue CCed him Pranith On 06/15/2014 07:16 PM, SINCOCK John wrote: > > Hi Everybody, > > We have recently setup a 2-node gluster, and are having some problems > deleting some files! > > We haven't accessed the gluster bricks directly, all access has been > through the glusterfs mount > > The 1^st node in our cluster also handles sharing the glusterfs volume > via samba, to windows clients. > > Environment is: > > Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC > 2013 x86_64 x86_64 x86_64 GNU/Linux > > CentOS release 6.4 (Final) > > glusterfs-libs-3.4.1-3.el6.x86_64 > > glusterfs-cli-3.4.1-3.el6.x86_64 > > glusterfs-fuse-3.4.1-3.el6.x86_64 > > glusterfs-server-3.4.1-3.el6.x86_64 > > glusterfs-3.4.1-3.el6.x86_64 > > 4 bricks on each node > > Gluster is setup without replication or striping > > Bricks on one of the two nodes (the other node is setup > the same): > > /dev/sda1 9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1 > > /dev/sda2 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2 > > /dev/sda3 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3 > > /dev/sda4 9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4 > > Gluster volume mounted on 1st node: > > g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00 > > Mount options: > > g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs > (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072) > > *So the problem is this:* > > [root@g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not > empty > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not > empty > > [root@g-unit-1 ~]# > > And the delete leaves behind 2000 or so files: > > [root@g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ > -type f | wc -l > > 2185 > > Almost all the left behind files are on the 2^nd node, g-unit-2, but > there are a few on g-unit-1 > > From the 2^nd node, g-unit-2: > > [root@g-unit-2 ~]# getfattr -m . -d -e hex > /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > getfattr: Removing leading '/' from absolute path names > > # file: > mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f > > So the file is there, is tagged with some gluster info, and has > definitely been copied onto the volume properly, via the glusterfs > mount. It hasn't been copied directly onto the brick. > > But from g-unit-1, accessing via the glusterfs mount, and trying to > delete this file: > > [root@g-unit-1 ~]# rm > /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > rm: remove regular file > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': > No such file or directory > > What the hell!?!!!!! > > Can someone please suggest the best course of action to resolve this > and prevent it happening in the future? > > Thanks for any assistance. > > John > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
