On 11/21/2013 07:54 PM, Alexandre Fournier wrote: > They are both regular file on the node and the replicas and they have the same GFID. I ran also the gluster volume heal gv0 split-brain command and the file is not in the list. We have an entire directory though (1023 entry on a node) > > However, the file was already on the brick before uploading it and I noticed that that the write did not work since the last modification date does not match the upload time. > > Through a web service, we offer to upload files on the gluster mount. This web service upload the file on a temporary folder and then MOVE the file on the gluster mount. > > Could the move operation give strange behavior like this? Alexandre, No, it should not. Please let us know the answers of the questions Pranith and I asked, so we can understand the root cause of your problem. > > Alexandre Fournier > Tools Programmer > Ubisoft Production Services > > > -----Original Message----- > From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] > Sent: 21 novembre 2013 00:47 > To: Lalatendu Mohanty > Cc: Alexandre Fournier; gluster-users at gluster.org; gluster-devel at nongnu.org > Subject: Re: [Gluster-devel] Self Heal and dangling symlinks > > Alexandre, > Seems like there is an entry split-brain (same file/dir name but on one brick it is a file and on the other it is a directory) according to the following log: >> [2013-11-18 18:18:43.052446] W >> [afr-common.c:1411:afr_conflicting_iattrs] >> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1) > Could you get us the output of "stat <brick-dir-path>/aa/aa/aa/aa/aa" and "getfattr -d -m. -e hex <brick-dir-path>/aa/aa/aa/aa/aa" on both the bricks. > > Pranith > ----- Original Message ----- >> From: "Lalatendu Mohanty" <lmohanty at redhat.com> >> To: "Alexandre Fournier" <alexandre.fournier at ubisoft.com>, >> gluster-users at gluster.org, gluster-devel at nongnu.org >> Sent: Thursday, November 21, 2013 1:28:01 AM >> Subject: Re: [Gluster-devel] Self Heal and dangling >> symlinks >> >> On 11/19/2013 10:49 PM, Alexandre Fournier wrote: >> >> >> >> >> >> Hello, >> >> >> >> We are experiencing strange behavior when writing file on the Gluster >> mount point. On some occasion, when writing to the Gluster Mount we >> have an Open Stream error. We?ve looked the gluster logs and found the >> following faulty entries : >> >> >> >> [From /var/log/glusterfs/mnt-gv0.log] >> >> >> >> [2013-11-18 18:18:43.052446] W >> [afr-common.c:1411:afr_conflicting_iattrs] >> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1) >> >> [2013-11-18 18:18:43.052468] E >> [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0: >> Conflicting entries for /aa/aa/aa/aa >> >> [2013-11-18 18:18:43.052757] E >> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] >> 0-gv0-replicate-0: background meta-data data entry missing-entry gfid >> self-heal >> >> failed on /aa/aa/aa/aa/aa >> >> [2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk] >> 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 >> (Input/output >> error) >> >> >> >> We?ve looked at the log file etc-glusterfs-glusterd.vol.log but we >> found nothing related to this problem. Then, we?ve looked at the log >> From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of >> logs of the same type : >> >> >> >> [2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft] >> 0-gv0-posix: symlink >> ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> >> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf >> failed (No such file or directory) >> >> [2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft] >> 0-gv0-posix: symlink >> ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 -> >> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 >> failed (No such file or directory) >> >> [2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft] >> 0-gv0-posix: symlink >> ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 -> >> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 >> failed (No such file or directory) >> >> >> >> This looks like a bug, unless there is something wrong with the >> set-up. I have copied gluster-devel in this thread as I think they might help. >> >> Just curious, is all your gluster nodes have equal time (i.e. ntp synced). >> >> >> >> >> >> >> And it does not stop logging. It seems that the self heal is not >> working properly when there are broking symlinks in the gluster. It is >> worth saying also that this log is only produce on a single node but >> the write fail on several node though. Also, we try to clean the >> symlinks manually but it always come back. >> >> >> >> Is it possible to recover from broken symlinks? >> >> >> >> Configuration : >> >> Gluster Version : 3.3.2 >> >> Cluster setup : 4 X 2 >> >> OS : Ubuntu >> >> On Fuse >> >> >> >> Thanks, >> >> >> >> Alexandre >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list Gluster-users at gluster.org >> http://supercolony.gluster.org/mailman/listinfo/gluster-users >> >> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at nongnu.org >> https://lists.nongnu.org/mailman/listinfo/gluster-devel >>
