Re: Rolling upgrades from glusterfs 3.4 to 3.5

Ravishankar N <ravishankar@xxxxxxxxxx> · Wed, 18 Jun 2014 17:27:56 +0530



    On 06/12/2014 11:13 PM, Anand Avati
      wrote:

    
          On Thu, Jun 12, 2014 at 10:33 AM,
            Vijay Bellur <vbellur@xxxxxxxxxx>
            wrote:

            
                On 06/12/2014 06:52 PM, Ravishankar N
                  wrote:

                  
                    Hi Vijay,

                    
                    Since glusterfs 3.5, posix_lookup() sends ESTALE
                    instead of ENOENT [1]

                    when when a parent gfid (entry) is not present on
                    the brick . In a

                    replicate set up, this causes a problem because AFR
                    gives more priority

                    to ESTALE than ENOENT, causing IO to fail [2]. The
                    fix is in progress at

                    [3] and is client-side specific , and would make it
                    to 3.5.2

                    
                    But we will still hit the problem when rolling
                    upgrade is performed from

                    3.4 to 3.5,  unless the clients are also upgraded to
                    3.5: To elaborate

                    an example:

                    
                    0) Create a 1x2 volume using 2 nodes and mount it
                    from client. All

                    machines are glusterfs 3.4

                    1) Perform for i in {1..30}; do mkdir $i; tar xf
                    glusterfs-3.5git.tar.gz

                    -C $i& done

                    2) While this is going on, kill one of the node in
                    the replica pair and

                    upgrade it to glusterfs 3.5 (simulating rolling
                    upgrade)

                    3) After a while, kill all tar processes

                    4) Create a backup directory and move all 1..30 dirs
                    inside 'backup'

                    5) Start the untar processes in 1) again

                    6) Bring up the upgraded node. Tar fails with estale
                    errors.

                    
                    Essentially the errors occur because [3] is a client
                    side fix. But

                    rolling upgrades are targeted at servers while the
                    older clients still

                    need to access them without issues.

                    
                    A solution is to have a fix in the posix translator
                    wherein the newer

                    client passes it's version (3.5) to posix_lookup()
                    which then sends

                    ESTALE if version is 3.5 or newer but sends ENOENT
                    instead if it is an

                    older client. Does this seem okay?

                    
              Cannot think of a better solution to this. Seamless
              rolling upgrades are necessary for us and the proposed fix
              does seem okay for that reason.

              
              Thanks,

              Vijay
              
                
            I also like Justin's proposal, of having fixes in 3.4.X
              and requiring clients to be at least 3.4.X in order to
              have rolling upgrade to 3.5.Y. This way we can add the
              "special fix" in 3.4.X client (just like the 3.5.2
              client). Ravi's proposal "works", but all LOOKUPs will
              have an extra xattr, and we will be carrying forward the
              compat code burden for a very long time. Whereas a 3.4.X
              client fix will remain in 3.4 branch.
            

            Thanks
            

    I have sent a fix for review (http://review.gluster.org/#/c/8080/) .
    The change is in the server side only. I reckon if we are asking
    users to upgrade clients to a 3.4.x  which anyway involves app
    downtime, we might as well ask them to upgrade to 3.5. 

    
    The fix is only sent on 3.5 - it does not need to go to master as I
    understand from Pranith that we only support compatibility between
    the current two releases. (meaning 3.6 servers require clients to be
    at at least 3.5 and not lower).

    
    Regards,

    Ravi

    
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-devel