On 11/21/2013 07:54 PM, Alexandre Fournier wrote:
They are both regular file on the node and the replicas and they have the same GFID. I ran also the gluster volume heal gv0 split-brain command and the file is not in the list. We have an entire directory though (1023 entry on a node)
However, the file was already on the brick before uploading it and I noticed that that the write did not work since the last modification date does not match the upload time.
Through a web service, we offer to upload files on the gluster mount. This web service upload the file on a temporary folder and then MOVE the file on the gluster mount.
Could the move operation give strange behavior like this?
Alexandre,
No, it should not. Please let us know the answers of the questions
Pranith and I asked, so we can understand the root cause of your problem.
Alexandre Fournier
Tools Programmer
Ubisoft Production Services
-----Original Message-----
From: Pranith Kumar Karampuri [mailto:pkarampu@xxxxxxxxxx]
Sent: 21 novembre 2013 00:47
To: Lalatendu Mohanty
Cc: Alexandre Fournier; gluster-users@xxxxxxxxxxx; gluster-devel@xxxxxxxxxx
Subject: Re: [Gluster-users] Self Heal and dangling symlinks
Alexandre,
Seems like there is an entry split-brain (same file/dir name but on one brick it is a file and on the other it is a directory) according to the following log:
[2013-11-18 18:18:43.052446] W
[afr-common.c:1411:afr_conflicting_iattrs]
0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)
Could you get us the output of "stat <brick-dir-path>/aa/aa/aa/aa/aa" and "getfattr -d -m. -e hex <brick-dir-path>/aa/aa/aa/aa/aa" on both the bricks.
Pranith
----- Original Message -----
From: "Lalatendu Mohanty" <lmohanty@xxxxxxxxxx>
To: "Alexandre Fournier" <alexandre.fournier@xxxxxxxxxxx>,
gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxx
Sent: Thursday, November 21, 2013 1:28:01 AM
Subject: Re: [Gluster-users] Self Heal and dangling
symlinks
On 11/19/2013 10:49 PM, Alexandre Fournier wrote:
Hello,
We are experiencing strange behavior when writing file on the Gluster
mount point. On some occasion, when writing to the Gluster Mount we
have an Open Stream error. We’ve looked the gluster logs and found the
following faulty entries :
[From /var/log/glusterfs/mnt-gv0.log]
[2013-11-18 18:18:43.052446] W
[afr-common.c:1411:afr_conflicting_iattrs]
0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)
[2013-11-18 18:18:43.052468] E
[afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0:
Conflicting entries for /aa/aa/aa/aa
[2013-11-18 18:18:43.052757] E
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
0-gv0-replicate-0: background meta-data data entry missing-entry gfid
self-heal
failed on /aa/aa/aa/aa/aa
[2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1
(Input/output
error)
We’ve looked at the log file etc-glusterfs-glusterd.vol.log but we
found nothing related to this problem. Then, we’ve looked at the log
From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of
logs of the same type :
[2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf ->
/mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf
failed (No such file or directory)
[2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 ->
/mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2
failed (No such file or directory)
[2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 ->
/mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2
failed (No such file or directory)
This looks like a bug, unless there is something wrong with the
set-up. I have copied gluster-devel in this thread as I think they might help.
Just curious, is all your gluster nodes have equal time (i.e. ntp synced).
And it does not stop logging. It seems that the self heal is not
working properly when there are broking symlinks in the gluster. It is
worth saying also that this log is only produce on a single node but
the write fail on several node though. Also, we try to clean the
symlinks manually but it always come back.
Is it possible to recover from broken symlinks?
Configuration :
Gluster Version : 3.3.2
Cluster setup : 4 X 2
OS : Ubuntu
On Fuse
Thanks,
Alexandre
_______________________________________________
Gluster-users mailing list Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
https://lists.nongnu.org/mailman/listinfo/gluster-devel
