Re: Problem with files showing zero links

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Joe Julian wrote on 28/01/2016 18:28:
> Response inline

Ditto. And thanks BTW. :-).

> On 01/28/2016 07:28 AM, Ronny Adsetts wrote:
>> Hi all,
>>
>> Have an issue I'm having trouble explaining or getting to the bottom of. I have a two node, two brick replicated 75G volume containing ~4661 files:
>>
>> gotham:~# gluster volume status software
>> Status of volume: software
>> Gluster process                                         Port    Online  Pid
>> ------------------------------------------------------------------------------
>> Brick gotham.stor.graysofwestminster.co.uk:/data/gluste
>> rfs/software/brick1/brick                               49152   Y       27296
>> Brick metropolis.stor.graysofwestminster.co.uk:/data/gl
>> usterfs/software/brick1/brick                           49152   Y       30335
>> NFS Server on localhost                                 2049    Y       27309
>> Self-heal Daemon on localhost                           N/A     Y       27316
>> NFS Server on metropolis.stor.graysofwestminster.co.uk  2049    Y       30348
>> Self-heal Daemon on metropolis.stor.graysofwestminster.
>> co.uk                                                   N/A     Y       30355
>>
>> Task Status of Volume software
>> ------------------------------------------------------------------------------
>> There are no active volume tasks
>>
>>
>> gotham:~# gluster volume heal software info
>> Brick gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
>> Number of entries: 0
>>
>> Brick metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
>> Number of entries: 0
>>
>>
>> I have the volume mounted on both nodes at /stor/software. On one of the nodes, there are lots of files showing with a link count of 0 which is leading to the Windows software using the data having validation issues:
>>
>> metropolis:~# ls -ltr /stor/software/win_patches/ | head
>> total 21585884
>> -rwxr--r-- 0 ainet Domain Admins     27816 Jan  4  2006 nullpatch.exe
>> -rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
>> -rwxrwxr-- 0 ronny Domain Admins    432568 Oct 29  2006 MSXML26SP3.exe
>> -rwxrwxr-- 0 ronny Domain Admins   1070592 Oct 29  2006 msxml3sp7.msi
>> -rwxrwxr-- 0 ronny Domain Admins    711160 Oct 29  2006 Windows2000-KB842773.EXE
>> -rwxrwxr-- 0 ronny Domain Admins    760824 Oct 29  2006 Windows2000-KB842933.EXE
>> -rwxrwxr-- 0 ronny Domain Admins   2585872 Oct 29  2006 WindowsInstaller-KB893803v2.exe
>> -rwxrwxr-- 0 ronny Domain Admins    106240 Oct 29  2006 Windows-KB870669.exe
>> -rwxrwxr-- 0 ronny Domain Admins   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>>
>>
>> From the other node:
>>
>> gotham:~# ls -ltr /stor/software/win_patches/ | head
>> total 21585884
>> -rwxr--r-- 1 1066 1016     27816 Jan  4  2006 nullpatch.exe
>> -rwxrwxr-- 1 1045 1016 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
>> -rwxrwxr-- 1 1045 1016    432568 Oct 29  2006 MSXML26SP3.exe
>> -rwxrwxr-- 1 1045 1016   1070592 Oct 29  2006 msxml3sp7.msi
>> -rwxrwxr-- 1 1045 1016    711160 Oct 29  2006 Windows2000-KB842773.EXE
>> -rwxrwxr-- 1 1045 1016    760824 Oct 29  2006 Windows2000-KB842933.EXE
>> -rwxrwxr-- 1 1045 1016   2585872 Oct 29  2006 WindowsInstaller-KB893803v2.exe
>> -rwxrwxr-- 1 1045 1016    106240 Oct 29  2006 Windows-KB870669.exe
>> -rwxrwxr-- 1 1045 1016   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>>
>>
>> I'm seeing the following and lots more similar in the logs:
>>
>> [2016-01-28 15:13:15.099478] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f)
> 
> client-1 is brick2, metropolis. That error says the file is missing, but it also shows the gfid. Perhaps the gfid hardlink is missing (the .glusterfs tree).
> 
> Check the metadata for the file on both bricks:
>    getfattr -m . -d -e hex /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> 
> Ensure the gfid is the same on both. Check the number of links on both, there should be 2. With a gifd of 489f648-9fde-4bcc-b906-6ef88ffcf90f, that file should be hard linked to .glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on both bricks.

"Good" brick:

gotham:~# getfattr -m . -d -e hex /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
trusted.afr.software-client-0=0x000000000000000000000000
trusted.afr.software-client-1=0x000000000000000000000000
trusted.gfid=0x3489f6489fde4bccb9066ef88ffcf90f

gotham:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

gotham:~# ls -i /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
114563677 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

gotham:~# find /data/glusterfs/software -inum 114563677
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
/data/glusterfs/software/brick1/brick/.glusterfs/34/89/3489f648-9fde-4bcc-b906-6ef88ffcf90f


"Bad" brick:

metropolis:~# getfattr -m . -d -e hex /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
trusted.afr.software-client-0=0x000000000000000000000000
trusted.afr.software-client-1=0x000000000000000000000000
trusted.gfid=0x3489f6489fde4bccb9066ef88ffcf90f

metropolis:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

metropolis:~# ls -i /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
132120540 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

metropolis:~# find /data/glusterfs/software -inum 132120540
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi


So we're missing the file in .glusterfs.


The magic question is I guess do you happen to know how to put the node back together?

Thanks.

Ronny

-- 
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957


Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux