What version of gluster are you using? Was there a chance there were
directory renames from the client?
There was a bug which Pranith fixed quite some time back:
http://review.gluster.org/#/c/7879/ for missing .glusterfs link files.
-Ravi
On 01/29/2016 05:58 PM, Venky Shankar wrote:
On Fri, Jan 29, 2016 at 11:34:37AM +0000, Ronny Adsetts wrote:
Venky Shankar wrote on 29/01/2016 11:09:
On Fri, Jan 29, 2016 at 10:46:14AM +0000, Ronny Adsetts wrote:
No idea how it came about in the first place. Suspect it happened
during either an operating system upgrade or upgrading glusterfs
packages to those from gluster.org.
If you're still willing to debug, please send across brick log file
of the concerned node.
Of course. Attached is the current brick log.
The log files does not have any clue about how some of the files ended up with missing .glusterfs
linkage, though there are tons of errors of operations trying to access .glusterfs/.././.....
(e.g. for gfid 7f0fa160-8e5d-44f5-88b3-8187b2397313). Probably an upgrade and restart of services
in midst of an operation?
My previous comments about having fixed the problem were a little premature. I now have most of the files that were showing with zero links now in the 'failed-heal' state:
# gluster volume heal software statistics
[...]
Starting time of crawl: Fri Jan 29 11:15:53 2016
Ending time of crawl: Fri Jan 29 11:15:58 2016
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 1976
My guess would be, as soon as you fixed the backend linkages, self-heal was able to kick in as it could
"find" these files on the replica. But why did the heal fail or why these files needed healing in the
first place.
Pranith, Ravi?
Also, please provide self-heal logs.
The volume log file shows all the details but the "E" error entries look like this:
[2016-01-29 11:12:00.479724] E [client-rpc-fops.c:914:client3_3_flush_cbk] 0-software-client-1: remote operation failed: Operation not permitted
And warnings like this:
[2016-01-29 11:11:33.747428] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/NDP20SP2-KB2446704-v2-x86.exe (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7)
[2016-01-29 11:11:33.749192] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: <gfid:9aa6b79a-b18f-42f2-ae9a-a886c523a0e7> (9aa6b79a-b18f-42f2-ae9a-a886c523a0e7)
Ronny
--
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com
Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
