On 06/22/2011 02:44 PM, Burnash, James wrote: > g01/pfs-ro1-client-0=0x000000000000000000000000 jc1letgfs17 > g01/pfs-ro1-client-0=0x000000000600000800000000 jc1letgfs18 > g01/pfs-ro1-client-20=0x000000000000000000000000 jc1letgfs14 > g01/pfs-ro1-client-20=0x000000000200000000000000 jc1letgfs15 > g02/pfs-ro1-client-2=0x000000000000000000000000 jc1letgfs17 > g02/pfs-ro1-client-2=0x000000004500000400000000 jc1letgfs18 > g02/pfs-ro1-client-22=0x000000000000000000000000 jc1letgfs14 > g02/pfs-ro1-client-22=0x000000000200000000000000 jc1letgfs15 > > Would anybody have any insights as to what is going on here? I'm > seeing attributes in my sleep these days ... that cannot be good! When I look at this, a few things occur to me. First, those are some pretty big metadata-change numbers. For g02 on fs18, 0x45000004 is actually 0x04000045 - about 67M - after byte swapping. The other thing that seems strange is that it always seems to be the second member of a replica pair "indicting" the first. Lastly, if you don't see any non-zero xattrs besides those above then this is not a normal split-brain situation. It might be a more exotic kind, based on *missing* xattrs. Here's the sequence: * Lookup for '/' on client-0 returns a zero opcounts for both client-0 and client-1 * Lookup for '/' on client-1 returns a non-zero opcount for client-0 and *no xattr at all* (i.e. missing) for client-1 * Client-1 is declared "ignorant" in afr_sh_build_pending_matrix * Client-0's value for client-1 is incremented, again in afr_sh_build_pending_matrix * Client-0 and client-1 are both marked "wise" in afr_sh_mark_sources * Client-0 and client-1 both receive wisdom=0 in afr_sh_compute_wisdom * No wisdom=1 nodes are found in afr_sh_wise_nodes_conflict * The "Unable to self-heal" messages come from afr_sh_metadata_fix So, what I'd do is check whether the following xattrs are missing: fs18/g01/pfs-ro1-client-1 fs15/g01/pfs-ro1-client-21 fs18/g02/pfs-ro1-client-3 fs15/g02/pfs-ro1-client-23 There might be other "false split-brain" scenarios lurking in this code. I don't fully understand it, but it seems to go through a lot of paths that might not have been fully tested.
