Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

[ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx>]

Hi Anuradha,

Please find the glusterfs and glusterd logs directory as an attachment.

Regards,

Abhishek

On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> wrote:

Hi Anuradha,

Sorry for late reply.

Please find the requested logs below:

Remote: 10.32.0.48
Local : 10.32.1.144

Local:
#gluster volume heal c_glusterfs info split-brain
Brick 10.32.1.144:/opt/lvmdir/c2/brick
Number of entries in split-brain: 0

Brick 10.32.0.48:/opt/lvmdir/c2/brick
Number of entries in split-brain: 0

Remote:
#gluster volume heal c_glusterfs info split-brain
Brick 10.32.1.144:/opt/lvmdir/c2/brick
Number of entries in split-brain: 0

Brick 10.32.0.48:/opt/lvmdir/c2/brick
Number of entries in split-brain: 0

auto-sync.sh.
Here you can see that i/o error is detected. Below is the required meta data from both the bricks.

1)
stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': Input/output error
Remote:

getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
# file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee

Local:

getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
# file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd


2)
stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error
Remote:

getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
# file: opt/lvmdir/c2/brick/security/corbasecurity
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
trusted.gfid=0xd298b7a0c8834f3e99abb39741363013

Local:

getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
# file: opt/lvmdir/c2/brick/security/corbasecurity
trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
trusted.gfid=0x890df0f706184b52803fac3242a2f15b

I observed that getfattr command output doesn't show all the fields all the times.

Here you can check that gluster split-brain command hasn't reported any split-brains but resulted in IO errors when accessed few files.

Could you please tell me if "split-brain" command doesn't reported any entry as output, then is there any way through which we can find out that the files are in split-brain if we are getting the IO error on those file.

Regards,

Abhishek

On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur@xxxxxxxxxx> wrote:

----- Original Message -----
> From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
> To: gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxxx
> Sent: Thursday, March 3, 2016 12:10:42 PM
> Subject: [Gluster-users] gluster volume heal info split brain command not     showing files in split-brain
>
>
> Hello,
>
> In gluster, we use the command "gluster volume heal c_glusterfs info
> split-brain" to find the files that are in split-brain scenario.
> We run heal script (developed by Windriver prime team) on the files reported
> by above command to resolve split-brain issue.
>
> But we observed that the above command is not showing all files that are in
> split-brain,
> even though split brain scenario actually exists on the node.
>
> Now a days this issue is seen more often and IO errors are reported when
> tried to access these files under split-brain.
>
> Can you please check why this gluster command is not showing files under
> split-brain?
> We can provide you required logs and support to resolve this issue.
Hi,

Could you paste the output of getfattr -m. -de hex <path-to-files-in-split-brain> from all the bricks that the files lie in?

>
> Please reply on this because I am not getting any reply from the community.
>
> --
>
> Regards
> Abhishek Paliwal
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> http://www.gluster.org/mailman/listinfo/gluster-users

--
Thanks,
Anuradha.

Attachment: glusterd.tar
Description: Unix tar archive

Attachment: glusterfs.tar
Description: Unix tar archive

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel