Hi Jeff, Thanks for your quick response, I have included results of the command below. > -----Original Message----- > From: gluster-users-bounces at gluster.org [mailto:gluster-users- > bounces at gluster.org] On Behalf Of Jeff Darcy > > The platform is not currently running production data and I have been > testing the redundancy of the setup (pulling cables etc.). All my > servers are now logging the following messages every 1 minute or so: > > > > [2010-11-11 14:18:49.636327] I [afr-common.c:672:afr_lookup_done] > datastore-replicate-0: split brain detected during lookup of /. > > [2010-11-11 14:18:49.636388] I [afr-common.c:716:afr_lookup_done] > datastore-replicate-0: background meta-data data self-heal triggered. > path: / > > [2010-11-11 14:18:49.636863] E [afr-self-heal- > metadata.c:524:afr_sh_metadata_fix] datastore-replicate-0: Unable to > self-heal permissions/ownership of '/' (possible split-brain). Please > fix the file on all backend volumes > > Can you do a "getfattr -d -e hex -m trusted.afr $path" on the path for > each brick's root directory (server side)? There seem to be a few > different ways for the split-brain flag to be set, all having to do > with > the contents of these xattrs. The solution might be to clear them, but > it would be good to see what the values are and have someone closer to > the AFR code than I am determine exactly which case we're in. [root at agent001 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary getfattr: Removing leading '/' from absolute path names # file: glusterfs/primary trusted.afr.datastore-client-0=0x000000000000000000000000 trusted.afr.datastore-client-1=0x000000000000000200000000 [root at agent001 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary getfattr: Removing leading '/' from absolute path names # file: glusterfs/secondary trusted.afr.datastore-client-4=0x000000000000000000000000 trusted.afr.datastore-client-5=0x000000000000000000000000 trusted.afr.datastore-client-6=0x000000000000000000000000 trusted.afr.datastore-client-7=0x000000000000000000000000 trusted.afr.datastore-client-8=0x000000000000000000000000 trusted.afr.datastore-client-9=0x000000000000000000000000 [root at agent002 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary getfattr: Removing leading '/' from absolute path names # file: glusterfs/primary trusted.afr.datastore-client-10=0x000000000000000000000000 trusted.afr.datastore-client-11=0x000000000000000000000000 trusted.afr.datastore-client-2=0x000000000000000000000000 trusted.afr.datastore-client-3=0x000000000000000000000000 trusted.afr.datastore-client-8=0x000000000000000000000000 trusted.afr.datastore-client-9=0x000000000000000200000000 [root at agent002 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary getfattr: Removing leading '/' from absolute path names # file: glusterfs/secondary trusted.afr.datastore-client-10=0x000000000000000300000000 trusted.afr.datastore-client-11=0x000000000000000000000000 trusted.afr.datastore-client-6=0x000000000000000000000000 trusted.afr.datastore-client-7=0x000000000000000000000000 trusted.afr.datastore-client-8=0x000000000000000000000000 trusted.afr.datastore-client-9=0x000000000000000000000000 [root at agent003 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary getfattr: Removing leading '/' from absolute path names # file: glusterfs/primary trusted.afr.datastore-client-2=0x000000000000000000000000 trusted.afr.datastore-client-3=0x000000000000000200000001 trusted.afr.datastore-client-4=0x000000000000000000000000 trusted.afr.datastore-client-5=0x000000000000000000000000 [root at agent003 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary getfattr: Removing leading '/' from absolute path names # file: glusterfs/secondary trusted.afr.datastore-client-0=0x000000000000000100000000 trusted.afr.datastore-client-1=0x000000000000000000000000 [root at agent004 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary getfattr: Removing leading '/' from absolute path names # file: glusterfs/primary trusted.afr.datastore-client-10=0x000000000000000300000000 trusted.afr.datastore-client-11=0x000000000000000000000000 trusted.afr.datastore-client-4=0x000000000000000000000000 trusted.afr.datastore-client-5=0x000000000000000500000000 trusted.afr.datastore-client-6=0x000000000000000000000000 trusted.afr.datastore-client-7=0x000000000000000000000000 trusted.afr.datastore-client-8=0x000000000000000000000000 trusted.afr.datastore-client-9=0x000000000000000000000000 [root at agent004 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary getfattr: Removing leading '/' from absolute path names # file: glusterfs/secondary trusted.afr.datastore-client-10=0x000000000000000000000000 trusted.afr.datastore-client-11=0x000000000000000000000000 trusted.afr.datastore-client-2=0x000000000000000000000000 trusted.afr.datastore-client-3=0x000000000000000000000000 trusted.afr.datastore-client-8=0x000000000000000000000000 trusted.afr.datastore-client-9=0x000000000000000200000000
