|
Response inline On 01/28/2016 07:28 AM, Ronny Adsetts
wrote:
Hi all, Have an issue I'm having trouble explaining or getting to the bottom of. I have a two node, two brick replicated 75G volume containing ~4661 files: gotham:~# gluster volume status software Status of volume: software Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick gotham.stor.graysofwestminster.co.uk:/data/gluste rfs/software/brick1/brick 49152 Y 27296 Brick metropolis.stor.graysofwestminster.co.uk:/data/gl usterfs/software/brick1/brick 49152 Y 30335 NFS Server on localhost 2049 Y 27309 Self-heal Daemon on localhost N/A Y 27316 NFS Server on metropolis.stor.graysofwestminster.co.uk 2049 Y 30348 Self-heal Daemon on metropolis.stor.graysofwestminster. co.uk N/A Y 30355 Task Status of Volume software ------------------------------------------------------------------------------ There are no active volume tasks gotham:~# gluster volume heal software info Brick gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/ Number of entries: 0 Brick metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/ Number of entries: 0 I have the volume mounted on both nodes at /stor/software. On one of the nodes, there are lots of files showing with a link count of 0 which is leading to the Windows software using the data having validation issues: metropolis:~# ls -ltr /stor/software/win_patches/ | head total 21585884 -rwxr--r-- 0 ainet Domain Admins 27816 Jan 4 2006 nullpatch.exe -rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29 2006 W2KA_SP4_128_x86_ENU.exe -rwxrwxr-- 0 ronny Domain Admins 432568 Oct 29 2006 MSXML26SP3.exe -rwxrwxr-- 0 ronny Domain Admins 1070592 Oct 29 2006 msxml3sp7.msi -rwxrwxr-- 0 ronny Domain Admins 711160 Oct 29 2006 Windows2000-KB842773.EXE -rwxrwxr-- 0 ronny Domain Admins 760824 Oct 29 2006 Windows2000-KB842933.EXE -rwxrwxr-- 0 ronny Domain Admins 2585872 Oct 29 2006 WindowsInstaller-KB893803v2.exe -rwxrwxr-- 0 ronny Domain Admins 106240 Oct 29 2006 Windows-KB870669.exe -rwxrwxr-- 0 ronny Domain Admins 1260024 Oct 29 2006 Windows2000-KB908506.EXE >From the other node: gotham:~# ls -ltr /stor/software/win_patches/ | head total 21585884 -rwxr--r-- 1 1066 1016 27816 Jan 4 2006 nullpatch.exe -rwxrwxr-- 1 1045 1016 135477136 Oct 29 2006 W2KA_SP4_128_x86_ENU.exe -rwxrwxr-- 1 1045 1016 432568 Oct 29 2006 MSXML26SP3.exe -rwxrwxr-- 1 1045 1016 1070592 Oct 29 2006 msxml3sp7.msi -rwxrwxr-- 1 1045 1016 711160 Oct 29 2006 Windows2000-KB842773.EXE -rwxrwxr-- 1 1045 1016 760824 Oct 29 2006 Windows2000-KB842933.EXE -rwxrwxr-- 1 1045 1016 2585872 Oct 29 2006 WindowsInstaller-KB893803v2.exe -rwxrwxr-- 1 1045 1016 106240 Oct 29 2006 Windows-KB870669.exe -rwxrwxr-- 1 1045 1016 1260024 Oct 29 2006 Windows2000-KB908506.EXE I'm seeing the following and lots more similar in the logs: [2016-01-28 15:13:15.099478] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f) client-1 is brick2, metropolis. That error says the file is missing, but it also shows the gfid. Perhaps the gfid hardlink is missing (the .glusterfs tree). Check the metadata for the file on both bricks: getfattr -m . -d -e hex /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi Ensure the gfid is the same on both. Check the number of links on both, there should be 2. With a gifd of 489f648-9fde-4bcc-b906-6ef88ffcf90f, that file should be hard linked to .glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on both bricks. File does exist on both bricks: metropolis:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi -rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi gotham:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi -rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi Any pointers to get to the bottom of this would be greatly appreciated. Look for log entries in the brick logs that corrospond to the timestamps on the client-side and see if there's some correlation. Thanks. Ronny |
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
