Hi Everybody, We have recently setup a 2-node gluster, and are having some problems deleting some files! We haven’t accessed the gluster bricks directly, all access has been through the glusterfs mount The 1st node in our cluster also handles sharing the glusterfs volume via samba, to windows clients. Environment is: Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux CentOS release 6.4 (Final) glusterfs-libs-3.4.1-3.el6.x86_64 glusterfs-cli-3.4.1-3.el6.x86_64 glusterfs-fuse-3.4.1-3.el6.x86_64 glusterfs-server-3.4.1-3.el6.x86_64 glusterfs-3.4.1-3.el6.x86_64 4 bricks on each node Gluster is setup without replication or striping Bricks on one of the two nodes (the other node is setup the same): /dev/sda1 9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1 /dev/sda2 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2 /dev/sda3 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3 /dev/sda4 9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4 Gluster volume mounted on 1st node: g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00 Mount options: g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072) So the problem is this: [root@g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not empty rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not empty [root@g-unit-1 ~]# And the delete leaves behind 2000 or so files: [root@g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f | wc -l 2185 Almost all the left behind files are on the 2nd node, g-unit-2, but there are a few on g-unit-1 From the 2nd node, g-unit-2: [root@g-unit-2 ~]# getfattr -m . -d -e hex /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi getfattr: Removing leading '/' from absolute path names # file: mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f So the file is there, is tagged with some gluster info, and has definitely been copied onto the volume properly, via the glusterfs mount. It hasn’t been copied directly onto the brick. But from g-unit-1, accessing via the glusterfs mount, and trying to delete this file: [root@g-unit-1 ~]# rm /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi rm: remove regular file `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': No such file or directory What the hell!?!!!!! Can someone please suggest the best course of action to resolve this and prevent it happening in the future? Thanks for any assistance. John |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
