That looks like you haven't had a split-brain since the 9th of October... On 11/21/2013 02:43 PM, Alexandre Fournier wrote: > I would like also to inform you about the information about the split-brain we have : > > 2013-10-09 22:02:59 <gfid:83b39e48-f8eb-4149-b851-d3b97e18c4b6> > 2013-10-09 21:52:59 <gfid:85378e3f-0dd1-4f8e-a7d5-70424b643fb9> > 2013-10-09 21:52:59 <gfid:0a958cad-1615-4e1b-8e1a-9dc0356859d6> > 2013-10-09 21:52:59 <gfid:54b02fde-69d2-4da2-8372-2a7af89a0ae1> > 2013-10-09 21:52:59 <gfid:4702c3ab-a2bb-43e3-ae2c-ecb5b440f368> > 2013-10-09 21:52:59 <gfid:8fe46824-a9f1-4095-b204-e9e137ae8643> > ... > Count : 1023 > > We tried to clean all the dangling links but they are still coming back the split-brain is not resolved. > > It maybe the root cause of the problem, how do we resolve those split brain? > > -----Original Message----- > From: Alexandre Fournier > Sent: 21 novembre 2013 15:20 > To: 'Lalatendu Mohanty'; Pranith Kumar Karampuri > Cc: gluster-users at gluster.org; gluster-devel at nongnu.org > Subject: RE: [Gluster-devel] Self Heal and dangling symlinks > > Ok here is the information : > > Stat : > File: `/aa/aa/aa/aa/aa/aa/aa > Size: 14364 Blocks: 32 IO Block: 4096 regular file > Device: 822h/2082d Inode: 3155137861 Links: 2 > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data) > Access: 2013-11-21 13:14:58.527765935 +0000 > Modify: 2013-11-13 13:19:13.736226050 +0000 > Change: 2013-11-13 13:19:13.736226050 +0000 > > File: `/aa/aa/aa/aa/aa/aa/aa > Size: 14364 Blocks: 32 IO Block: 4096 regular file > Device: 822h/2082d Inode: 3076494286 Links: 2 > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data) > Access: 2013-11-21 13:14:58.527674754 +0000 > Modify: 2013-11-13 13:19:13.736442464 +0000 > Change: 2013-11-13 13:19:13.736442464 +0000 > Birth: - > > Attributes : > > # file: aa/aa/aa/aa/aa/aa/aa > trusted.afr.gv0-client-0=0x000000000000000000000000 > trusted.afr.gv0-client-1=0x000000000000000000000000 > trusted.gfid=0xb5b8c3ec9dd24609b56476651113d3fa > > > # file: aa/aa/aa/aa/aa/aa/aa > trusted.afr.gv0-client-0=0x000000000000000000000000 > trusted.afr.gv0-client-1=0x000000000000000000000000 > trusted.gfid=0xb5b8c3ec9dd24609b56476651113d3fa > > > -----Original Message----- > From: Lalatendu Mohanty [mailto:lmohanty at redhat.com] > Sent: 21 novembre 2013 14:05 > To: Alexandre Fournier; Pranith Kumar Karampuri > Cc: gluster-users at gluster.org; gluster-devel at nongnu.org > Subject: Re: [Gluster-devel] Self Heal and dangling symlinks > > On 11/21/2013 07:54 PM, Alexandre Fournier wrote: >> They are both regular file on the node and the replicas and they have >> the same GFID. I ran also the gluster volume heal gv0 split-brain >> command and the file is not in the list. We have an entire directory >> though (1023 entry on a node) >> >> However, the file was already on the brick before uploading it and I noticed that that the write did not work since the last modification date does not match the upload time. >> >> Through a web service, we offer to upload files on the gluster mount. This web service upload the file on a temporary folder and then MOVE the file on the gluster mount. >> >> Could the move operation give strange behavior like this? > Alexandre, > > No, it should not. Please let us know the answers of the questions Pranith and I asked, so we can understand the root cause of your problem. > >> Alexandre Fournier >> Tools Programmer >> Ubisoft Production Services >> >> >> -----Original Message----- >> From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] >> Sent: 21 novembre 2013 00:47 >> To: Lalatendu Mohanty >> Cc: Alexandre Fournier; gluster-users at gluster.org; >> gluster-devel at nongnu.org >> Subject: Re: [Gluster-devel] Self Heal and dangling >> symlinks >> >> Alexandre, >> Seems like there is an entry split-brain (same file/dir name but on one brick it is a file and on the other it is a directory) according to the following log: >>> [2013-11-18 18:18:43.052446] W >>> [afr-common.c:1411:afr_conflicting_iattrs] >>> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, >>> 1) >> Could you get us the output of "stat <brick-dir-path>/aa/aa/aa/aa/aa" and "getfattr -d -m. -e hex <brick-dir-path>/aa/aa/aa/aa/aa" on both the bricks. >> >> Pranith >> ----- Original Message ----- >>> From: "Lalatendu Mohanty" <lmohanty at redhat.com> >>> To: "Alexandre Fournier" <alexandre.fournier at ubisoft.com>, >>> gluster-users at gluster.org, gluster-devel at nongnu.org >>> Sent: Thursday, November 21, 2013 1:28:01 AM >>> Subject: Re: [Gluster-devel] Self Heal and dangling >>> symlinks >>> >>> On 11/19/2013 10:49 PM, Alexandre Fournier wrote: >>> >>> >>> >>> >>> >>> Hello, >>> >>> >>> >>> We are experiencing strange behavior when writing file on the Gluster >>> mount point. On some occasion, when writing to the Gluster Mount we >>> have an Open Stream error. We?ve looked the gluster logs and found >>> the following faulty entries : >>> >>> >>> >>> [From /var/log/glusterfs/mnt-gv0.log] >>> >>> >>> >>> [2013-11-18 18:18:43.052446] W >>> [afr-common.c:1411:afr_conflicting_iattrs] >>> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, >>> 1) >>> >>> [2013-11-18 18:18:43.052468] E >>> [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0: >>> Conflicting entries for /aa/aa/aa/aa >>> >>> [2013-11-18 18:18:43.052757] E >>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] >>> 0-gv0-replicate-0: background meta-data data entry missing-entry gfid >>> self-heal >>> >>> failed on /aa/aa/aa/aa/aa >>> >>> [2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk] >>> 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 >>> (Input/output >>> error) >>> >>> >>> >>> We?ve looked at the log file etc-glusterfs-glusterd.vol.log but we >>> found nothing related to this problem. Then, we?ve looked at the log >>> From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of >>> logs of the same type : >>> >>> >>> >>> [2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft] >>> 0-gv0-posix: symlink >>> ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> >>> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf >>> failed (No such file or directory) >>> >>> [2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft] >>> 0-gv0-posix: symlink >>> ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 -> >>> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 >>> failed (No such file or directory) >>> >>> [2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft] >>> 0-gv0-posix: symlink >>> ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 -> >>> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 >>> failed (No such file or directory) >>> >>> >>> >>> This looks like a bug, unless there is something wrong with the >>> set-up. I have copied gluster-devel in this thread as I think they might help. >>> >>> Just curious, is all your gluster nodes have equal time (i.e. ntp synced). >>> >>> >>> >>> >>> >>> >>> And it does not stop logging. It seems that the self heal is not >>> working properly when there are broking symlinks in the gluster. It >>> is worth saying also that this log is only produce on a single node >>> but the write fail on several node though. Also, we try to clean the >>> symlinks manually but it always come back. >>> >>> >>> >>> Is it possible to recover from broken symlinks? >>> >>> >>> >>> Configuration : >>> >>> Gluster Version : 3.3.2 >>> >>> Cluster setup : 4 X 2 >>> >>> OS : Ubuntu >>> >>> On Fuse >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Alexandre >>> >>> >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list Gluster-users at gluster.org >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> _______________________________________________ >>> Gluster-devel mailing list >>> Gluster-devel at nongnu.org >>> https://lists.nongnu.org/mailman/listinfo/gluster-devel >>> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users
