Hi Anuradha,
I am doing the same which is mentioned in the link you shared and It has been resolved the issue.
But my question is if it is the split-brain scenario then why the command "gluster volume heal info split-brain"
not showing these files in the output even not the parent directory is present in split-brain.
Please find the requested logs
On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur <atalur@xxxxxxxxxx> wrote:
Hi Abhishek,
The files that are reporting i/o error have gfid-mismatch. This situation is called directory or entry split-brain. You can find steps to resolve this kind of split brain here : https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ .
Ideally, the parent directories of these files have to be listed in heal info split-brain output. Can you please get extended attributes of parent directories of the files that show i/o error (Same getfattr command that you previously used.) ?
----- Original Message -----
> From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
--> To: "Anuradha Talur" <atalur@xxxxxxxxxx>
> Cc: gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxxx
> Sent: Thursday, March 10, 2016 11:22:35 AM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain
>
> Hi Anuradha,
>
> Please find the glusterfs and glusterd logs directory as an attachment.
>
> Regards,
> Abhishek
>
>
>
> On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx>
> wrote:
>
> > Hi Anuradha,
> >
> > Sorry for late reply.
> >
> > Please find the requested logs below:
> >
> > Remote: 10.32.0.48
> > Local : 10.32.1.144
> >
> > Local:
> > #gluster volume heal c_glusterfs info split-brain
> > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Remote:
> > #gluster volume heal c_glusterfs info split-brain
> > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > Number of entries in split-brain: 0
> >
> > auto-sync.sh.
> > Here you can see that i/o error is detected. Below is the required meta
> > data from both the bricks.
> >
> > 1)
> > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> > Input/output error
> > Remote:
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > trusted.afr.dirty=0x000000000000000000000000
> > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee
> >
> > Local:
> >
> > getfattr -d -m . -e hex
> > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd
> >
> >
> > 2)
> > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error
> > Remote:
> >
> > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > trusted.afr.dirty=0x000000000000000000000000
> > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013
> >
> > Local:
> >
> > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > trusted.gfid=0x890df0f706184b52803fac3242a2f15b
> >
> > I observed that getfattr command output doesn't show all the fields all
> > the times.
> >
> > Here you can check that gluster split-brain command hasn't reported any
> > split-brains but resulted in IO errors when accessed few files.
> >
> > Could you please tell me if "split-brain" command doesn't reported any
> > entry as output, then is there any way through which we can find out that
> > the files are in split-brain if we are getting the IO error on those file.
> >
> >
> > Regards,
> > Abhishek
> >
> >
> >
> > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur@xxxxxxxxxx> wrote:
> >
> >>
> >>
> >> ----- Original Message -----
> >> > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx>
> >> > To: gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxxx
> >> > Sent: Thursday, March 3, 2016 12:10:42 PM
> >> > Subject: [Gluster-users] gluster volume heal info split brain command
> >> not showing files in split-brain
> >> >
> >> >
> >> > Hello,
> >> >
> >> > In gluster, we use the command "gluster volume heal c_glusterfs info
> >> > split-brain" to find the files that are in split-brain scenario.
> >> > We run heal script (developed by Windriver prime team) on the files
> >> reported
> >> > by above command to resolve split-brain issue.
> >> >
> >> > But we observed that the above command is not showing all files that
> >> are in
> >> > split-brain,
> >> > even though split brain scenario actually exists on the node.
> >> >
> >> > Now a days this issue is seen more often and IO errors are reported when
> >> > tried to access these files under split-brain.
> >> >
> >> > Can you please check why this gluster command is not showing files under
> >> > split-brain?
> >> > We can provide you required logs and support to resolve this issue.
> >> Hi,
> >>
> >> Could you paste the output of getfattr -m. -de hex
> >> <path-to-files-in-split-brain> from all the bricks that the files lie in?
> >>
> >> >
> >> > Please reply on this because I am not getting any reply from the
> >> community.
> >> >
> >> > --
> >> >
> >> > Regards
> >> > Abhishek Paliwal
> >> >
> >> > _______________________________________________
> >> > Gluster-users mailing list
> >> > Gluster-users@xxxxxxxxxxx
> >> > http://www.gluster.org/mailman/listinfo/gluster-users
> >>
> >> --
> >> Thanks,
> >> Anuradha.
> >>
> >
>
Thanks,
Anuradha.
--
Regards
Abhishek Paliwal
1.File Name : nameroot.ior stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': Input/output error Remote File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x000000000000000256e812da0007bf13 trusted.gfid=0xc18f775d94de42879235d1331d85c860 Remote Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files # file: opt/lvmdir/c2/brick/public_html/cello/ior_files trusted.afr.c_glusterfs-client-1=0x000000000000000000000000 trusted.afr.c_glusterfs-client-207=0x000000000000000000000002 trusted.afr.c_glusterfs-client-209=0x000000000000000000000000 trusted.afr.c_glusterfs-client-215=0x000000000000000000000000 trusted.afr.c_glusterfs-client-39=0x000000000000000000000000 trusted.afr.c_glusterfs-client-47=0x000000000000000000000000 trusted.afr.c_glusterfs-client-49=0x000000000000000000000002 trusted.afr.c_glusterfs-client-51=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xd9cd3be03fa44d1e8a8da8523535ef0a trusted.glusterfs.dht=0x000000010000000000000000ffffffff stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': Input/output error Local File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior trusted.bit-rot.version=0x000000000000000256e813c50008b4e2 trusted.gfid=0x32145e0378864767989335f37c108409 Local Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files # file: opt/lvmdir/c2/brick/public_html/cello/ior_files trusted.afr.c_glusterfs-client-112=0x000000000000000000000000 trusted.afr.c_glusterfs-client-116=0x000000000000000000000000 trusted.afr.c_glusterfs-client-128=0x000000000000000000000000 trusted.afr.c_glusterfs-client-130=0x000000000000000000000000 trusted.afr.c_glusterfs-client-150=0x000000000000000000000000 trusted.afr.c_glusterfs-client-164=0x000000000000000000000000 trusted.afr.c_glusterfs-client-166=0x000000000000000000000000 trusted.afr.c_glusterfs-client-194=0x000000000000000000000000 trusted.afr.c_glusterfs-client-196=0x000000000000000000000000 trusted.afr.c_glusterfs-client-200=0x000000000000000000000000 trusted.afr.c_glusterfs-client-224=0x000000000000000000000000 trusted.afr.c_glusterfs-client-26=0x000000000000000000000000 trusted.afr.c_glusterfs-client-36=0x000000000000000000000000 trusted.afr.c_glusterfs-client-38=0x000000000000000000000000 trusted.afr.c_glusterfs-client-40=0x000000000000000000000000 trusted.afr.c_glusterfs-client-50=0x000000000000000000000000 trusted.afr.c_glusterfs-client-54=0x000000000000000000000000 trusted.afr.c_glusterfs-client-58=0x000000000000000000000002 trusted.afr.c_glusterfs-client-64=0x000000000000000000000000 trusted.afr.c_glusterfs-client-66=0x000000000000000000000000 trusted.afr.c_glusterfs-client-70=0x000000000000000000000000 trusted.afr.c_glusterfs-client-76=0x000000000000000000000000 trusted.afr.c_glusterfs-client-84=0x000000000000000000000000 trusted.afr.c_glusterfs-client-90=0x000000000000000000000000 trusted.afr.c_glusterfs-client-98=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xd9cd3be03fa44d1e8a8da8523535ef0a trusted.glusterfs.dht=0x000000010000000000000000ffffffff ********************************************************************* 2.File Name : corbasecurity stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error Remote File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity # file: opt/lvmdir/c2/brick/security/corbasecurity trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x000000000000000256e812da0007bf13 trusted.gfid=0xe477a482ed354795a83b00281d85ef69 Remote Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security # file: opt/lvmdir/c2/brick/security trusted.afr.c_glusterfs-client-1=0x000000000000000000000000 trusted.afr.c_glusterfs-client-207=0x000000000000000000000002 trusted.afr.c_glusterfs-client-209=0x000000000000000000000000 trusted.afr.c_glusterfs-client-215=0x000000000000000000000000 trusted.afr.c_glusterfs-client-39=0x000000000000000000000000 trusted.afr.c_glusterfs-client-47=0x000000000000000000000000 trusted.afr.c_glusterfs-client-49=0x000000000000000000000000 trusted.afr.c_glusterfs-client-51=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x69eee88cddb64c27b99048123191b112 trusted.glusterfs.dht=0x000000010000000000000000ffffffff stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error Local File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity # file: opt/lvmdir/c2/brick/security/corbasecurity trusted.bit-rot.version=0x000000000000000256e813c50008b4e2 trusted.gfid=0x23400a9240ce422a99e24a09d7da8f6c Local Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security # file: opt/lvmdir/c2/brick/security trusted.afr.c_glusterfs-client-112=0x000000000000000000000000 trusted.afr.c_glusterfs-client-116=0x000000000000000000000000 trusted.afr.c_glusterfs-client-128=0x000000000000000000000000 trusted.afr.c_glusterfs-client-130=0x000000000000000000000000 trusted.afr.c_glusterfs-client-150=0x000000000000000000000000 trusted.afr.c_glusterfs-client-164=0x000000000000000000000000 trusted.afr.c_glusterfs-client-166=0x000000000000000000000000 trusted.afr.c_glusterfs-client-194=0x000000000000000000000000 trusted.afr.c_glusterfs-client-196=0x000000000000000000000000 trusted.afr.c_glusterfs-client-200=0x000000000000000000000000 trusted.afr.c_glusterfs-client-224=0x000000000000000000000000 trusted.afr.c_glusterfs-client-26=0x000000000000000000000000 trusted.afr.c_glusterfs-client-36=0x000000000000000000000000 trusted.afr.c_glusterfs-client-38=0x000000000000000000000000 trusted.afr.c_glusterfs-client-40=0x000000000000000000000000 trusted.afr.c_glusterfs-client-50=0x000000000000000000000000 trusted.afr.c_glusterfs-client-54=0x000000000000000000000000 trusted.afr.c_glusterfs-client-58=0x000000000000000000000002 trusted.afr.c_glusterfs-client-64=0x000000000000000000000000 trusted.afr.c_glusterfs-client-66=0x000000000000000000000000 trusted.afr.c_glusterfs-client-70=0x000000000000000000000000 trusted.afr.c_glusterfs-client-76=0x000000000000000000000000 trusted.afr.c_glusterfs-client-84=0x000000000000000000000000 trusted.afr.c_glusterfs-client-90=0x000000000000000000000000 trusted.afr.c_glusterfs-client-98=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x69eee88cddb64c27b99048123191b112 trusted.glusterfs.dht=0x000000010000000000000000ffffffff
_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
