> This is a standard problem where there are split-brains in distributed > systems. For example even in git there are cases where it gives up asking > users to fix the file i.e. merge conflicts. If the user doesn't want > split-brains they should move to replica-3 and enable client-quorum. But if > the user made a conscious decision to live with split-brain problems > favouring availability/using replica-2, then split-brains do happen and it > needs user intervention. All we are trying to do is to make this process a > bit painless by coming up with meaningful policies. > Agreed, split brains do require manual intervention no one argues about that, but it shouldn't be quite as tedious as GlusterFS wants it to be. I do agree that it is way simpler than perhaps some other distributed filesystems but at any point we ask some one to write a script to fix our internal structure - that is not a feature its a bug. We all appreciate the effort, but my wish we incorporate some pain points which we have seen personally over the years and fix it right when we are at it. > If the user knows his workload is append only and there are split-brains the > only command he needs to execute is: > 'gluster volume heal <volname> split-brain bigger-file' > no grep, no finding file paths, nothing. > Adding to this - we need to provide additional sanity checks that split brains were indeed fixed - since this looks quite destructive operation, are you planning a rollback at any point during this process? > There were also instances where the user knows the brick he/she would like > to be the source but he/she is worried that old brick which comes back up > would cause split-brains so he/she had to erase the whole brick which was > down and bring it back up. > Instead we can suggest him/her to use 'gluster volume heal <VOLNAME> > split-brain source-brick <brick_name>' after bringing the brick back up so > that not all the contents needs to be healed. > 1) gluster volume heal <volname> info split-brain should give output in some > 'format' giving stat/pending-matrix etc for all the files in split-brain. > - Unfortunately we still don't have a way to provide with file paths > without doing 'find' on the bricks. Critical setups require fixing split-brain with quick turn around no one really has the luxury running a find on a large volume. So i still do not understand, if a 'find' can do a gfid --> inum --> path - how hard it is for Gluster management daemon to know this? just to provide better tooling? -- Harsha _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-devel
