Thanks Franco, that's interesting, I've read some more emails to this list now, and yes this does look like the same issue that other people are seeing. So I hope those patches will fix the problem. I am not in any hurry to upgrade to 3.5 considering there still seem to be some other issues with 3.5 - but this seems to be quite a serious issue, if files copied to a gluster are not accessible if their parent directory did not end up on the longest-lived node! I'll have to keep a close eye on this issue and upgrade as soon as the 3.5 looks stable Thanks very much for the info. Cheers and Regards, John -----Original Message----- From: Franco Broi [mailto:franco.broi@xxxxxxxxxx] Sent: Wednesday, 18 June 2014 3:15 PM To: SINCOCK John Cc: gluster-users@xxxxxxxxxxx Subject: Re: Unable to delete files but getfattr shows file is part of glusterfs Hi John I got this yesterday, it was copied to this list. Cheers, On Tue, 2014-06-17 at 04:55 -0400, Susant Palai wrote: Hi Franco: > The following patches address the ENOTEMPTY issue. > > 1. http://review.gluster.org/#/c/7733/ > 2. http://review.gluster.org/#/c/7599/ > > I think the above patches will be available in 3.5.1 which will be a > minor upgrade.(Need ack from Niels de Vos.) > On Wed, 2014-06-18 at 13:26 +0930, SINCOCK John wrote: > Thanks Pranith, I hope Susant can help! > > > > From: Pranith Kumar Karampuri [mailto:pkarampu@xxxxxxxxxx] > Sent: Wednesday, 18 June 2014 11:48 AM > To: SINCOCK John; gluster-users@xxxxxxxxxxx > Cc: Susant Palai > Subject: Re: Unable to delete files but getfattr shows > file is part of glusterfs > > > > > I think Susant knows about this issue > > CCed him > > Pranith > > On 06/15/2014 07:16 PM, SINCOCK John wrote: > > > Hi Everybody, > > > > We have recently setup a 2-node gluster, and are having some > problems deleting some files! > > We haven’t accessed the gluster bricks directly, all access > has been through the glusterfs mount > > The 1st node in our cluster also handles sharing the glusterfs > volume via samba, to windows clients. > > > > Environment is: > > Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 > 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > > CentOS release 6.4 (Final) > > > > glusterfs-libs-3.4.1-3.el6.x86_64 > > glusterfs-cli-3.4.1-3.el6.x86_64 > > glusterfs-fuse-3.4.1-3.el6.x86_64 > > glusterfs-server-3.4.1-3.el6.x86_64 > > glusterfs-3.4.1-3.el6.x86_64 > > > > 4 bricks on each node > > Gluster is setup without replication or > striping > > > > Bricks on one of the two nodes (the other node > is setup the same): > > /dev/sda1 9.1T 4.7T 4.5T > 51% /mnt/glusterfs/bricks/1 > > /dev/sda2 9.1T 3.7T 5.5T > 40% /mnt/glusterfs/bricks/2 > > /dev/sda3 9.1T 3.7T 5.5T > 40% /mnt/glusterfs/bricks/3 > > /dev/sda4 9.1T 4.1T 5.0T > 46% /mnt/glusterfs/bricks/4 > > > > Gluster volume mounted on 1st node: > > g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00 > > Mount options: > > g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs > > (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=1310 > 72) > > > > So the problem is this: > > [root@g-unit-1 ~]# rm > -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': > Directory not empty > > rm: cannot remove > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': > Directory not empty > > [root@g-unit-1 ~]# > > > > And the delete leaves behind 2000 or so files: > > [root@g-unit-1 ~]# > find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f | > wc -l > > 2185 > > Almost all the left behind files are on the 2nd node, > g-unit-2, but there are a few on g-unit-1 > > > > From the 2nd node, g-unit-2: > > [root@g-unit-2 ~]# getfattr -m . -d -e > hex > /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64 > .efi > > getfattr: Removing leading '/' from absolute path names > > # file: > > mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64. > efi > > trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f > > So the file is there, is tagged with some gluster info, and > has definitely been copied onto the volume properly, via the > glusterfs mount. It hasn’t been copied directly onto the > brick. > > > > But from g-unit-1, accessing via the glusterfs mount, and > trying to delete this file: > > [root@g-unit-1 ~]# > rm > /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi > > rm: remove regular file > > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y > > rm: cannot remove > > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': > No such file or directory > > > > What the hell!?!!!!! > > > > Can someone please suggest the best course of action to > resolve this and prevent it happening in the future? > > > > Thanks for any assistance. > > > > John > > > > > > > > > > > > > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
