Re: [Gluster-devel] cannot delete non-empty directory

"David F. Robinson" <david.robinson@xxxxxxxxxxxxx> · Mon, 09 Feb 2015 17:57:40 +0000

> Yes. This will solve the issue you are facing now. After this Aegis 
can be removed from the mount point.
Understood.  Thanks for the clarification.

I looked at one of the zero length files in .glusterfs:

[root@gfs01bkp 54]# ls -al 2f54d764-6215-4dd0-bbcb-fef2bfedc7d8
---------T 2 rbhinge pme_ics 0 Jan 22 10:12 
2f54d764-6215-4dd0-bbcb-fef2bfedc7d8

[root@gfs01bkp 54]# getfattr -d -m . -e text 
2f54d764-6215-4dd0-bbcb-fef2bfedc7d8
# file: 2f54d764-6215-4dd0-bbcb-fef2bfedc7d8
trusted.gfid="/T×dbMÐ»Ëþò¿íÇØ"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

> A dump of all the xattrs matching regex "trusted.glusterfs.*" should 
list all the xattrs. The value of "trusted.glusterfs.dht.linkto" xattr 
should give the destination subvolume.
> If the file is not present on the destination, then its a stale linkto 
file pointing to non-existent file (on destination subvol) and it can be 
removed.
> Otherwise they are valid and shouldn't be removed.

Looks like the first one I checked is a stale link.

> Again as shyam mentioned in previous mail [1] should've fixed the 
issue (it is present in v3.6.0 and above). Not sure why we are seeing 
this issue again.

Once you figure this out, do you or will you have some kind of tool to 
go through and clean up all of these stale links?  Or, would you just 
leave them as they are?

David

------ Original Message ------
From: "Raghavendra Gowdappa" <rgowdapp@xxxxxxxxxx>
To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Cc: "Shyam" <srangana@xxxxxxxxxx>; "Gluster Devel" 
<gluster-devel@xxxxxxxxxxx>; gluster-users@xxxxxxxxxxx; "Susant Palai" 
<spalai@xxxxxxxxxx>
Sent: 2/9/2015 12:50:28 PM
Subject: Re: [Gluster-devel] cannot delete non-empty directory

----- Original Message -----
 From: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
 To: "Shyam" <srangana@xxxxxxxxxx>, "Gluster Devel" 
<gluster-devel@xxxxxxxxxxx>, gluster-users@xxxxxxxxxxx, "Susant
 Palai" <spalai@xxxxxxxxxx>
 Sent: Monday, February 9, 2015 10:55:44 PM
 Subject: Re: [Gluster-devel] cannot delete non-empty directory

 So, just to be sure before I do this, it is okay to do the following 
if
 I want to get rid of everything in the /old_shelf4/Aegis directory 
and
 below?

   rm -rf /data/brick*/homegfs_bkp/backup.0/old_shelf4/Aegis

Yes. This will solve the issue you are facing now. After this Aegis can 
be removed from the mount point.

 What happens to all of the files in the .glusterfs directory? Does 
this
 get rebuilt or do the links stay there for files that now no longer
 exist?

Links stay there for files that now no longer exist. This is not an 
issue except that we'll be loosing an inode (no data-blocks as file 
size was 0).

 And, is this same issue what causes all of the broken links in
 .glusterfs. See attached image for example. There appears to be a lot
 of broken links the .glusterfs directories. Is this normal or does it
 indicate another problem.

There can be other issues which can result in links not getting deleted 
from .glusterfs directory. Current issue is not related to that.

 Finally, if I search through the /data/brick* directories, should I 
find
 no entries of "-------T" permission files with zero length files? Do 
I
 need to clean all of these up somehow? A quick look at
 /data/brick01bkp/homegfs_bkp/.glusterfs/2f/54 shows many of these 
files.
   They look like
 ---------T 3 rbhinge pme_ics 0 Jan 9 16:45
 2f54d7d6-968b-442f-8cfe-eff01d6cefe7
 ---------T 2 rbhinge pme_ics 0 Jan 9 21:40
 2f54d7e7-b198-4fd4-aec7-f5d0ff020f72

 How do I find out what file these entries were pointing to?

As shyam had mentioned in an earlier mail, these files represent dht 
"linkto" files. These are sort of metadata containing the name of the 
subvolume where actual file is stored (hence the name "link-to"). The 
destination to which this "linkto" is pointing is stored in xattrs. A 
dump of all the xattrs matching regex "trusted.glusterfs.*" should list 
all the xattrs. The value of "trusted.glusterfs.dht.linkto" xattr 
should give the destination subvolume. If the file is not present on 
the destination, then its a stale linkto file pointing to non-existent 
file (on destination subvol) and it can be removed. Otherwise they are 
valid and shouldn't be removed.

Again as shyam mentioned in previous mail [1] should've fixed the issue 
(it is present in v3.6.0 and above). Not sure why we are seeing this 
issue again.

[1] http://review.gluster.org/8602

 David

 ------ Original Message ------
 From: "Shyam" <srangana@xxxxxxxxxx>
 To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>; "Gluster 
Devel"
 <gluster-devel@xxxxxxxxxxx>; "gluster-users@xxxxxxxxxxx"
 <gluster-users@xxxxxxxxxxx>; "Susant Palai" <spalai@xxxxxxxxxx>
 Sent: 2/9/2015 11:11:20 AM
 Subject: Re: [Gluster-devel] cannot delete non-empty directory

 >On 02/08/2015 12:19 PM, David F. Robinson wrote:
 >>I am seeing these messsages after I delete large amounts of data 
using
 >>gluster 3.6.2.
 >>cannot delete non-empty directory:

>>old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
 >>*_From the FUSE mount (as root), the directory shows up as empty:_*
 >># pwd

>>/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
 >>
 >># ls -al
 >>total 5
 >>d--------- 2 root root 4106 Feb 6 13:55 .
 >>drwxrws--- 3 601 dmiller 72 Feb 6 13:55 ..
 >>However, when you look at the bricks, the files are still there 
(none
 >>on
 >>brick01bkp, all files are on brick02bkp). All of the files are
 >>0-length
 >>and have ------T permissions.
 >
 >These files are linkto files that are created by DHT, which 
basically
 >mean the files were either renamed, or the brick layout changed (I
 >suspect the former to be the cause).
 >
 >These files should have been deleted when the files that they point 
to
 >were deleted, looks like this did not happen.
 >
 >Can I get the following information for some of the files here,
 >- getfattr -d -m . -e text <path to file on brick>
 > - The output of trusted.glusterfs.dht.linkto xattr should state 
where
 >the real file belongs, in this case as there are only 2 bricks, it
 >should be brick01bkp subvol
 >- As the second brick is empty, we should be able to safely delete
 >these files from the brick and proceed to do an rmdir on the mount
 >point of the volume as the directory is now empty.
 >- Please check, the one sub-directory that is showing up in this 
case
 >as well, "save1"
 >
 >>Any suggestions on how to fix this and how to prevent it from
 >>happening?
 >
 >I believe there are renames happening here, possibly by the archive
 >creator, one way to prevent the rename from creating a linkto file 
is
 >to use the DHT set parameter to set a pattern so that file name hash
 >considers only the static part of the name.
 >
 >The set parameter is, cluster.extra-hash-regex.
 >
 >A link on a similar problem and how to use this set parameter (there 
a
 >few in the gluster forums) would be,

>http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html
 >
 >Additionally, there is a bug here, the unlink of the file should 
have
 >cleaned up the linkto as well, so that all of the above is not
 >required, we have noticed this with NFS and FUSE mounts (ref bugs,
 >1117923, 1139992), and investigation is in progress on the same. We
 >will step up the priority on this so that we have a clean fix that 
can
 >be used to prevent this in the future.
 >
 >Shyam
 _______________________________________________
 Gluster-devel mailing list
 Gluster-devel@xxxxxxxxxxx
 http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users