Emmanuel, What do the extended attributes look like in the backend for this file on the two bricks. Pranith. ----- Original Message ----- > From: "Emmanuel Dreyfus" <manu@xxxxxxxxxx> > To: "Gluster Devel" <gluster-devel@xxxxxxxxxx> > Sent: Friday, June 21, 2013 6:23:31 AM > Subject: split brain with all-zero pending matrix > > On 3.4.0beta3, after using the volume for a while, I get split-brain errors > with an unhelpful pending matrix. > > [2013-06-20 08:44:00.665731] E > [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] > 0-gfs34-replicate-1: Unable to self-heal contents of > '/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o' (possible > split-brain). Please delete the file from all but the preferred subvolume.- > Pending matrix: [ [ 0 0 ] [ 0 0 ] ] > [2013-06-20 08:44:00.666431] E > [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] > 0-gfs34-replicate-1: background data self-heal failed on > /manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o > [2013-06-20 08:44:00.667201] W [afr-open.c:213:afr_open] 0-gfs34-replicate-1: > failed to open as split brain seen, returning EIO > [2013-06-20 08:44:00.668193] W [fuse-bridge.c:875:fuse_fd_cbk] > 0-glusterfs-fuse: 8711927: OPEN() > /manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o => -1 (Input/output > error) > > On the bricks (ls -l, md5 and first 32 bytes): > brick0 > -rw-r--r-- 2 manu manu 7216 Jun 20 09:56 > /export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o > MD5 (/export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) = > 24ee57aa8e2aeb6102ba170fb81bbf22 > 00000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 > |.ELF............| > 00000010 01 00 03 00 01 00 00 00 00 00 00 00 00 00 00 00 > |................| > brick1 > -rw-r--r-- 2 manu manu 7216 Jun 20 09:56 > /export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o > MD5 (/export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) = > 24ee57aa8e2aeb6102ba170fb81bbf22 > 00000000 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 > |.ELF............| > 00000010 01 00 03 00 01 00 00 00 00 00 00 00 00 00 00 00 > |................| > brick2 > -rw-r--r-- 2 manu manu 6256 Jun 20 09:56 > /export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o > MD5 (/export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) = > 58a4b8c5929cac2f799c60b3dd2acc2f > brick3 > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > -rw-r--r-- 2 manu manu 7216 Jun 20 09:56 > /export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o > MD5 (/export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) = > 2bc383747432c49793a2cf1a2c0e8cfa > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > > That shows the offending copies without any doubt, but I still wonder how > we come to that situation. There are a lot of such files, making the > glusterfs > volume just unusable at this point. > > -- > Emmanuel Dreyfus > http://hcpnet.free.fr/pubz > manu@xxxxxxxxxx > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxx > https://lists.nongnu.org/mailman/listinfo/gluster-devel >
