Re: [Gluster-users] gluster 3.7.9 permission denied and mv errors

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Raghavendra,

Last night the backup had four of these errors and only one of the 'retried moves' succeeded. The only one to succeed in moving the file the second time had target files on a different gluster peer (gfs01bkp). Not sure if that is significant.

Note that I cannot stat the target file over the FUSE mount for any of these, but it exists on the bricks. Running an 'ls' on the directory containing the file (via FUSE) does not fix the issue. Source and target xattrs are appended for all bricks on all machines in the distributed volume.

Let me know if there's any other information it would be useful to gather, as this issue seems to recur frequently.

Thanks,
Patrick

# Move failures

/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/056-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/090-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/057-2/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/54/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4': File exists

/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/056-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/090-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/057-2/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/54/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4': File exists

========================================================================================================================
retry: renaming ./homegfs/hpc_shared/motorsports/056-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3

source xattrs
  gfs01bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
 
  gfs02bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3
      trusted.bit-rot.version=0x0200000000000000570308980001d157
      trusted.gfid=0xe07abd8ae861442ebc0df8b20719af30
      trusted.pgfid.1776adb6-2925-49d3-9cca-8a04c29f4c05=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory

target xattrs
  gfs01bkp
   getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
   getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
 
  gfs02bkp
    # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3
      trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00
      trusted.gfid=0xaefffbd0676649cd95eb6dfc874d7a59
      trusted.pgfid.f7c5eff3-f474-433b-b10e-480f8353c6b9=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3
      trusted.gfid=0xaefffbd0676649cd95eb6dfc874d7a59
      trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200
      trusted.pgfid.f7c5eff3-f474-433b-b10e-480f8353c6b9=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory
 
stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3"': No such file or directory
 
Rename Failed! ./homegfs/hpc_shared/motorsports/056-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3
 
========================================================================================================================
retry: renaming ./homegfs/hpc_shared/motorsports/090-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3
 
source xattrs
  gfs01bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory

  gfs02bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3
      trusted.bit-rot.version=0x0200000000000000570308980001d157
      trusted.gfid=0x93663986a2d84632b0b358289458add2
      trusted.pgfid.15ff8972-e668-42f0-9254-cf5b5679e372=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
   
target xattrs
  gfs01bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory

  gfs02bkp
    # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3
      trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00
      trusted.gfid=0x830250ea6c08472e9d71a032818d4f9b
      trusted.pgfid.3254d2bf-69e0-4931-8c1b-d41cd0b2eff7=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3
      trusted.gfid=0x830250ea6c08472e9d71a032818d4f9b
      trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200
      trusted.pgfid.3254d2bf-69e0-4931-8c1b-d41cd0b2eff7=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory
 
stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3"': No such file or directory
 
Rename Failed! ./homegfs/hpc_shared/motorsports/090-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3
========================================================================================================================
retry: renaming ./homegfs/hpc_shared/motorsports/057-2/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3
 
source xattrs
  gfs01bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
   
  gfs02bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3
      trusted.bit-rot.version=0x0200000000000000570308980001d157
      trusted.gfid=0xa91444cec80d49cd98cbe53be4d5377e
      trusted.pgfid.eee7bcf2-4f21-444d-a470-c37071e4d7ab=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
   
target xattrs
  gfs01bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory

  gfs02bkp
    # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3
      trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00
      trusted.gfid=0xdf95487705d34b2592b5eaed1e8dc4d3
      trusted.pgfid.24a52e78-6e7c-4536-92b1-ee4419323e54=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3
      trusted.gfid=0xdf95487705d34b2592b5eaed1e8dc4d3
      trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200
      trusted.pgfid.24a52e78-6e7c-4536-92b1-ee4419323e54=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory
 
stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3"': No such file or directory
 
Rename Failed! ./homegfs/hpc_shared/motorsports/057-2/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3
========================================================================================================================
retry: renaming ./homegfs/hpc_shared/motorsports/54/data_collected4 -> ../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4
 
source xattrs
  gfs01bkp
    # file: data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4
      trusted.bit-rot.version=0x02000000000000005702ecdc0001e78e
      trusted.gfid=0x7860691318554b82a0f3ab8dd90fea98
      trusted.pgfid.e3906384-e16a-4a5f-9b07-f156558954db=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
   
  gfs02bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
   
target xattrs
  gfs01bkp
    # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4
      trusted.gfid=0x980c12097507431d953ee458ec14ca4a
      trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3100
      trusted.pgfid.6a8ecf7c-5597-4725-9764-455f7e267667=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
    # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4
      trusted.bit-rot.version=0x0200000000000000569bb8d700074a42
      trusted.gfid=0x980c12097507431d953ee458ec14ca4a
      trusted.pgfid.6a8ecf7c-5597-4725-9764-455f7e267667=0x00000001
   
    getfattr: Removing leading '/' from absolute path names
   
  gfs02bkp
    getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
    getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory
 
stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4"': No such file or directory
Rename Succeeded!
========================================================================================================================
 



On Fri, Apr 29, 2016 at 10:21 AM, Glomski, Patrick <patrick.glomski@xxxxxxxxxxxxx> wrote:
Raghavendra,

This error is occurring in a shell script moving files between directories on a FUSE mount when overwriting an old file with a newer file (it's a backup script, moving an incremental backup of a file into a 'rolling full backup' directory).

As a temporary workaround, we parse the output of this shell script for move errors and handle the errors as they happen. Simply re-moving the files fails, so we stat the destination (to see if we can learn anything about the type of file that causes this behavior), delete the destination, and try the move again (success!). Typical output is as follows:

/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4': File exists
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4': File exists
  File: `../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4'
  Size: 1714            Blocks: 4          IO Block: 131072 regular file
Device: 13h/19d Inode: 11051758947722304158  Links: 1
Access: (0660/-rw-rw----)  Uid: (  628/pkeistler)   Gid: ( 2020/   gmirl)
Access: 2016-01-20 17:20:45.000000000 -0500
Modify: 2015-11-06 15:20:41.000000000 -0500
Change: 2016-01-27 03:35:00.434712146 -0500
retry: renaming ./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4 -> ../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p11/149/data_collected4

Not sure if that description rings any bells as to what the problem might be, but if not, I added some code to print out the 'getattr' for the source and destination file on all of the bricks (before we delete the destination) and will post to this thread the next time we have that issue.

Thanks,
Patrick


On Fri, Apr 29, 2016 at 8:15 AM, Raghavendra G <raghavendra@xxxxxxxxxxx> wrote:


On Wed, Apr 13, 2016 at 10:00 PM, David F. Robinson <david.robinson@xxxxxxxxxxxxx> wrote:
I am running into two problems (possibly related?).   

1) Every once in a while, when I do a 'rm -rf DIRNAME', it comes back with an error: 
        rm: cannot remove `DIRNAME` : Directory not empty

        If I try the 'rm -rf' again after the error, it deletes the directory.  The issue is that I have scripts that clean up directories, and they are failing unless I go through the deletes a 2nd time.

What kind of mount are you using? Is it a FUSE or NFS mount? Recently we saw a similar issue on NFS clients on RHEL6 where rm -rf used to fail with ENOTEMPTY in some specific cases.
 

2) I have different scripts to move a large numbers of files (5-25k) from one directory to another.  Sometimes I receive an error: 
    /bin/mv: cannot move `xyz` to `../bkp00/xyz`: File exists

Does ./bkp00/xyz exist on backend? If yes, what is the value of gfid xattr (key: "trusted.gfid") for "xyz" and "./bkp00/xyz" on backend bricks (I need gfid from all the bricks) when this issue happens?


    The move is done using '/bin/mv -f', so it should overwrite the file if it exists.  I have tested this with hundreds of files, and it works as expected.  However, every few days the script that moves the files will have problems with 1 or 2 files during the move.  This is one move problem out of roughly 10,000 files that are being moved and I cannot figure out any reason for the intermittent problem.

Setup details for my gluster configuration shown below.

[root@gfs01bkp logs]# gluster volume info
 
Volume Name: gfsbackup
Type: Distribute
Volume ID: e78d5123-d9bc-4d88-9c73-61d28abf0b41
Status: Started
Number of Bricks: 7
Transport-type: tcp
Bricks:
Brick1: gfsib01bkp.corvidtec.com:/data/brick01bkp/gfsbackup
Brick2: gfsib01bkp.corvidtec.com:/data/brick02bkp/gfsbackup
Brick3: gfsib02bkp.corvidtec.com:/data/brick01bkp/gfsbackup
Brick4: gfsib02bkp.corvidtec.com:/data/brick02bkp/gfsbackup
Brick5: gfsib02bkp.corvidtec.com:/data/brick03bkp/gfsbackup
Brick6: gfsib02bkp.corvidtec.com:/data/brick04bkp/gfsbackup
Brick7: gfsib02bkp.corvidtec.com:/data/brick05bkp/gfsbackup
Options Reconfigured:
nfs.disable: off
server.allow-insecure: on
storage.owner-gid: 100
server.manage-gids: on
cluster.lookup-optimize: on
server.event-threads: 8
client.event-threads: 8
changelog.changelog: off
storage.build-pgfid: on
performance.readdir-ahead: on
diagnostics.brick-log-level: WARNING
diagnostics.client-log-level: WARNING
cluster.rebal-throttle: aggressive
performance.cache-size: 1024MB
performance.write-behind-window-size: 10MB


[root@gfs01bkp logs]# rpm -qa | grep gluster
glusterfs-server-3.7.9-1.el6.x86_64
glusterfs-debuginfo-3.7.9-1.el6.x86_64
glusterfs-api-3.7.9-1.el6.x86_64
glusterfs-resource-agents-3.7.9-1.el6.noarch
gluster-nagios-common-0.1.1-0.el6.noarch
glusterfs-libs-3.7.9-1.el6.x86_64
glusterfs-fuse-3.7.9-1.el6.x86_64
glusterfs-extra-xlators-3.7.9-1.el6.x86_64
glusterfs-geo-replication-3.7.9-1.el6.x86_64
glusterfs-3.7.9-1.el6.x86_64
glusterfs-cli-3.7.9-1.el6.x86_64
glusterfs-devel-3.7.9-1.el6.x86_64
glusterfs-rdma-3.7.9-1.el6.x86_64
samba-vfs-glusterfs-4.1.11-2.el6.x86_64
glusterfs-client-xlators-3.7.9-1.el6.x86_64
glusterfs-api-devel-3.7.9-1.el6.x86_64
python-gluster-3.7.9-1.el6.noarch


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel



--
Raghavendra G

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux