undeleteble directory

tetSU <tetsucceed@xxxxxxxxx> · Thu, 1 Dec 2016 12:12:44 +0400

Hello
i have some strange problem that happens on one of the servers with glusterfs.

Configuration:

-- gluster volume info --

Volume Name: gisdata
Type: Distribute
Volume ID: a49bcb69-fcc5-4f39-b556-e23e29da38e7
Status: Started
Number of Bricks: 7
Transport-type: tcp
Bricks:
Brick1: 10.1.0.4:/export/sdc1/brick
Brick2: 10.1.0.5:/export/sdc1/brick
Brick3: 10.1.0.6:/export/sdc1/brick
Brick4: 10.1.0.7:/export/sdc1/brick
Brick5: 10.1.0.9:/export/sdc1/brick
Brick6: 10.1.0.10:/export/sdc1/brick
Brick7: 10.1.0.12:/export/sdc1/brick
Options Reconfigured:
server.statedump-path: /tmp
performance.cache-size: 512MB
nfs.drc: off

System mount to master server as  

10.1.0.4:/gisdata on /usr/local/mountpoint type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

and its share by samba to the client who manage files on distribute filesystem.

Client tryes to delete directory (Data/Projects/DELIVERY/D39) but it fails with error: Can't delete, directory is not empty.
When i inspect that directory on the Bricks i found it totaly broken

 ls -lah /export/sdc1/brick/Data/Projects/DELIVERY/D39/
ls: cannot access /export/sdc1/brick/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO: No such file or directory
total 16K
drwxr-xr-x  3 root 500 8.0K Dec  1 10:52 .
drwxr-xr-x 40 root 500 4.0K Nov 30 22:43 ..
??????????  ? ?    ?      ?            ? 494-11-23-1_WD_PO

When i recreate that directory it looks fine but has doubled:

ls -lai /export/sdc1/brick/Data/Projects/DELIVERY/D39/
total 16
  30075838 drwxr-xr-x  4 root  500 8192 Dec  1 10:56 .
   9185992 drwxr-xr-x 40 root  500 4096 Nov 30 22:43 ..
2147631741 drwxr-xr-x  2 root root    6 Dec  1 10:56 494-11-23-1_WD_PO
2147631741 drwxr-xr-x  2 root root    6 Dec  1 10:56 494-11-23-1_WD_PO

Last logs:

/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.352895] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.355764] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.358448] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.381921] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.843840] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.846111] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.848384] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.857213] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.678051] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.680209] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.682785] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.697692] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.880597] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.885190] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.889059] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.905035] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.985591] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.990133] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.995346] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:49.012616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.278130] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.283321] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.286403] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.305513] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 09:52:26.319979] E [dht-rebalance.c:1496:gf_defrag_fix_layout] 0-gisdata-dht: /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO lookup failed
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.172294] I [dht-layout.c:640:dht_layout_normalize] 0-gisdata-dht: found anomalies in /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO. holes=1 overlaps=0
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.173616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO

It happens on server 10.1.0.9. Any ideas how to fix that?

-- 
_________________________
- tetsucceed

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users