Re: Split brain directory

[Karthik Subrahmanya <ksubrahm@xxxxxxxxxx>]

Hey,

From the getfattr output you have provided, the directory is clearly not in split brain.

If all the bricks are being blamed by others then it is called split brain.
In your case only client-13 that is Brick-14 in the volume info output had a pending entry heal on the directory.

That is the last replica subvol which consists of the bricks

Brick13: glusterserver03.mydomain.local:/bricks/video/brick3/safe
Brick14: glusterserver04.mydomain.local:/bricks/video/brick3/safe
Brick15: glusterserver05.mydomain.local:/bricks/video/brick3/safe (arbiter)

Which got healed as part of the heal you ran, or part of the self heal crawl and pending xattrs got reset to all zeros.

Which file are you not able to access? Can you give the getfattr output of that file and give the shd log
and the mount log where you were not able to access the file.

Regards,

Karthik

On Wed, Jan 24, 2018 at 2:00 PM, Luca Gervasi <luca.gervasi@xxxxxxxxx> wrote:

Hello,

I'm trying to fix an issue with a Directory Split on a gluster 3.10.3. The effect consist of a specific file in this splitted directory to randomly be unavailable on some clients.

I have gathered all the informations on this gist: https://gist.githubusercontent.com/lucagervasi/534e0024d349933eef44615fa8a5c374/raw/52ff8dd6a9cc8ba09b7f258aa85743d2854f9acc/splitinfo.txt

I discovered the splitted directory by the extended attributes (lines 172,173, 291,292, 

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.vol-video-client-13=0x000000000000000000000000

Seen on the bricks 

* /bricks/video/brick3/safe/video.mysite.it/htdocs/ su glusterserver05 (lines 278 ro 294)

* /bricks/video/brick3/safe/video.mysite.it/htdocs/ su glusterserver03 (lines 159 to 175)

Reading the documentation about afr extended attributes, this situation seems unclear (Docs from [1] and [2])

as own changelog is 0, same as client-13 (glusterserver02.mydomain.local:/bricks/video/brick3/safe)

as my understanding, such "dirty" attributes seems to indicate no split at all (feel free to correct me).

Some days ago, I issued a "gluster volume heal vol-video full", which endend (probably) that day, leaving no info on /var/log/gluster/glustershd.log nor fixing this split.

I tried to trigger a self heal using "stat" and "ls -l" over the splitted directory from a glusterfs mounted client directory, without having the bit set cleared.

The volume heal info split-brain itself shows zero items to be healed (lines 388 to 446).

All the clients mount this volume using glusterfs-fuse.

I don't know what to do, please help.

Thanks.

Luca Gervasi

References:

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/Recovering_from_File_Split-brain.html

[2] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/sect-managing_split-brain

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users