Re: [Gluster-users] Fencing FOPs on data-split-brained files

Ravishankar N <ravishankar@xxxxxxxxxx> · Tue, 19 Nov 2013 16:03:14 +0530



    On 11/16/2013 01:42 AM, Anand Avati
      wrote:

    
        Ravi,
        We should not mix up data and entry operation domains, if a
          file is in data split brain that should not stop a user from
          rename/link/unlink operations on the file.
        

          Regarding your concern about complications while healing - we
          should change our "manual fixing" instructions to:
        

        - go to backend, access through gfid path or normal path
        - rmxattr the afr changelogs
        - truncate the file to 0 bytes (like "> filename")
        

        Accessing the path through gfid and truncating to 0 bytes
          addresses your concerns about hardlinks/renames.
        

        Avati
      
      
    All,

      
      I
      have tabulated what operations must/ musn't be permitted in case
      of
      different split brains. Some of the columns are '?' as I am not
      sure
      what the expected behaviour should be. Could we have this
      validated? 

    
          File
                Operation permitted
          Type of Split
                Brain
        
        
          Data SB
          Metadata SB
          Entry SB
        
        
          Same entry gfid
                mismatch SB
          Different entries
        
        
          write
          No
          Yes (currently no)
          No
          Yes
        
        
          read
          No
          Yes (currently no)
          No
          Yes
        
        
          getfattr
          Yes
          No
          No
          Yes
        
        
          lookup
          ?
          ?
          No
          Yes
        
        
          stat/fstat
          ?
          ?
          No
          Yes
        
        
          setfattr
          Yes
          No
          No
          Yes
        
        
          touch
          Yes
          Yes
          No
          Yes
        
        
          hard link
              creation
          Yes
          Yes
          No
          Yes
        
        
          soft link
              creation
          Yes
          Yes
          Yes
          Yes
        
        
          rename
          Yes
          Yes
          no
          Yes
        
        
          chown
          Yes
          Yes
          Currently No
          Yes
        
        
          chmod
          Yes
          Yes
          Currently No
          Yes
        
        
          unlink
          Yes
          Yes
          Currently No
          Yes
        
        
          readdir
          N/A
          N/A
          ?
          ?
        
      
        - stat() also reports the file size. If a data split-brained
        file has different sizes, should stat succeed?

        - Likewise if metadata split brain is due to different access
        permissions, say one brick has file chmod'ed with 777 and the
        other brick has it with 744, should we allow read/write if the
        corresponding permission bits are *not* conflciting ? ( as of
        today they aren't allowed)

      
      Also,In the table above, Entry Split brain has 2 cases- 

        i) where same entry has different gfids

        ii) each brick  has different entries for the same directory
        (which can cause deleted files to appear in case of conservative
        merge). 

        Should we allow readdir in either case?

        
        Thanks,

        Ravi

      
        On Wed, Nov 13, 2013 at 3:01 AM,
          Ravishankar N <ravishankar@xxxxxxxxxx>
          wrote:

          Hi,

            
            Currenly in glusterfs, when there is a data splt-brain
            (only) on a file, we disallow the following operations from
            the mount-point by returning EIO to the application:

            - Writes to the file (truncate, dd, echo, cp etc)

            - Reads to the file (cat)

            - Reading extended attributes (getfattr) [1]

            
            However we do permit the following operations:

            -creating hardlinks

            -creating symlinks

            -mv

            -setattr

            -chmod

            -chown

            --touch

            -ls

            -stat

            
            While it makes sense to allow `ls` and `stat`, is it okay to
             add checks in the FOPS to disallow the other operations?
            Allowing creation of links and changing file attributes only
            seems to complicate things before the admin can go to the
            backend bricks and resolve the splitbrain (by deleteing all
            but the healthy copy of the file including hardlinks). More
            so if the file is renamed before addressing the split-brain.

            Please share your thoughs.

            
            Thanks,

            Ravi

            
            [1] http://review.gluster.org/#/c/5988/

            _______________________________________________

            Gluster-users mailing list

            Gluster-users@xxxxxxxxxxx

            http://supercolony.gluster.org/mailman/listinfo/gluster-users