Hi Abhishek, The files that are reporting i/o error have gfid-mismatch. This situation is called directory or entry split-brain. You can find steps to resolve this kind of split brain here : https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ . Ideally, the parent directories of these files have to be listed in heal info split-brain output. Can you please get extended attributes of parent directories of the files that show i/o error (Same getfattr command that you previously used.) ? ----- Original Message ----- > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > To: "Anuradha Talur" <atalur@xxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxxx > Sent: Thursday, March 10, 2016 11:22:35 AM > Subject: Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain > > Hi Anuradha, > > Please find the glusterfs and glusterd logs directory as an attachment. > > Regards, > Abhishek > > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <abhishpaliwal@xxxxxxxxx> > wrote: > > > Hi Anuradha, > > > > Sorry for late reply. > > > > Please find the requested logs below: > > > > Remote: 10.32.0.48 > > Local : 10.32.1.144 > > > > Local: > > #gluster volume heal c_glusterfs info split-brain > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Remote: > > #gluster volume heal c_glusterfs info split-brain > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > Number of entries in split-brain: 0 > > > > auto-sync.sh. > > Here you can see that i/o error is detected. Below is the required meta > > data from both the bricks. > > > > 1) > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': > > Input/output error > > Remote: > > > > getfattr -d -m . -e hex > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > trusted.afr.dirty=0x000000000000000000000000 > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee > > > > Local: > > > > getfattr -d -m . -e hex > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd > > > > > > 2) > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error > > Remote: > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > trusted.afr.dirty=0x000000000000000000000000 > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 > > > > Local: > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b > > > > I observed that getfattr command output doesn't show all the fields all > > the times. > > > > Here you can check that gluster split-brain command hasn't reported any > > split-brains but resulted in IO errors when accessed few files. > > > > Could you please tell me if "split-brain" command doesn't reported any > > entry as output, then is there any way through which we can find out that > > the files are in split-brain if we are getting the IO error on those file. > > > > > > Regards, > > Abhishek > > > > > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur@xxxxxxxxxx> wrote: > > > >> > >> > >> ----- Original Message ----- > >> > From: "ABHISHEK PALIWAL" <abhishpaliwal@xxxxxxxxx> > >> > To: gluster-users@xxxxxxxxxxx, gluster-devel@xxxxxxxxxxx > >> > Sent: Thursday, March 3, 2016 12:10:42 PM > >> > Subject: [Gluster-users] gluster volume heal info split brain command > >> not showing files in split-brain > >> > > >> > > >> > Hello, > >> > > >> > In gluster, we use the command "gluster volume heal c_glusterfs info > >> > split-brain" to find the files that are in split-brain scenario. > >> > We run heal script (developed by Windriver prime team) on the files > >> reported > >> > by above command to resolve split-brain issue. > >> > > >> > But we observed that the above command is not showing all files that > >> are in > >> > split-brain, > >> > even though split brain scenario actually exists on the node. > >> > > >> > Now a days this issue is seen more often and IO errors are reported when > >> > tried to access these files under split-brain. > >> > > >> > Can you please check why this gluster command is not showing files under > >> > split-brain? > >> > We can provide you required logs and support to resolve this issue. > >> Hi, > >> > >> Could you paste the output of getfattr -m. -de hex > >> <path-to-files-in-split-brain> from all the bricks that the files lie in? > >> > >> > > >> > Please reply on this because I am not getting any reply from the > >> community. > >> > > >> > -- > >> > > >> > Regards > >> > Abhishek Paliwal > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users@xxxxxxxxxxx > >> > http://www.gluster.org/mailman/listinfo/gluster-users > >> > >> -- > >> Thanks, > >> Anuradha. > >> > > > -- Thanks, Anuradha. _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
