Re: [gluster] possible split-brain issue

Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx> · Fri, 23 Jan 2015 13:11:53 +0530



    hi Arnold,

       You gave the output only on one brick it seems? Could you also
    provide it on other brick as well. Sorry I didn't make that clear in
    my earlier mail.

    
    Pranith

    On 01/23/2015 10:44 AM, Arnold Yang
      wrote:

    
        Hi Pranith,
         
        Here is the
            output for the commands provide by you, anything more you
            need, please tell us!
        Thanks!
         
        [root@dmf-wpst-1
            ~]# getfattr -d -m. -e hex /export/vdb1/brick/
        getfattr:
            Removing leading '/' from absolute path names
        # file:
            export/vdb1/brick/
        security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
        trusted.afr.gv0-client-0=0x000000000000000000000000
        trusted.afr.gv0-client-1=0x000000000000001400000000
        trusted.gfid=0x00000000000000000000000000000001
        trusted.glusterfs.dht=0x000000010000000000000000ffffffff
        trusted.glusterfs.volume-id=0x51de44c3f01e486da6b710c7b7a270d7
         
         
        [root@dmf-wpst-1
            ~]# getfattr -d -m. -e hex /export/vdb1/brick/mpdis/
        getfattr:
            Removing leading '/' from absolute path names
        # file:
            export/vdb1/brick/mpdis/
        security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
        trusted.afr.gv0-client-0=0x000000000000000000000000
        trusted.afr.gv0-client-1=0x000000000000000400000000
        trusted.gfid=0x8ff7afeb996244cd9d1bf213568398d7
        trusted.glusterfs.dht=0x000000010000000000000000ffffffff
         
        [root@dmf-wpst-1
            ~]# getfattr -d -m. -e hex
            /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1
        getfattr:
            Removing leading '/' from absolute path names
        # file:
            export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf1
        security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
        trusted.afr.gv0-client-0=0x000000000000000000000000
        trusted.afr.gv0-client-1=0x000000000000000000000000
        trusted.gfid=0x85ed306b179b46819d7c02eb336543b8
         
         
        [root@dmf-wpst-1
            ~]# getfattr -d -m. -e hex
            /export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2
        getfattr:
            Removing leading '/' from absolute path names
        # file:
            export/vdb1/brick/mpdis/test.rep.00.00.00.00.dmf2
        security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
        trusted.afr.gv0-client-0=0x000000000000000000000000
        trusted.afr.gv0-client-1=0x000000000000000000000000
        trusted.gfid=0xa826a389e7a042c2b5175a1acbecae9b
         
         
            From:
                Pranith Kumar Karampuri [mailto:pkarampu@xxxxxxxxxx]
                

                Sent: Thursday, January 22, 2015 12:14 AM

                To: Jifeng Li; Gluster-users@xxxxxxxxxxx; Arnold
                Yang

                Subject: Re:  [gluster] possible
                split-brain issue
          
        
          On 01/14/2015 04:48 PM, Jifeng Li wrote:
        
        
          Hi ,
           
          [issue]:
             To
              ensure the glusterFS mount point work, a script will
              periodically using HTTP put a file to subdirectory under
              mount point which is used as  Apache DocumentRoot. But
              after running some time,  some errors show below:
          [2015-01-14 09:18:40.915639] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /mpdis
          [2015-01-14 09:18:41.924584] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 20 ] [ 21 0
            ] ]
          [2015-01-14 09:18:41.925182] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /
          [2015-01-14 09:18:41.934827] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 4 ] [ 2 0 ]
            ]
          [2015-01-14 09:18:41.935375] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /mpdis
          [2015-01-14 09:18:42.943742] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 20 ] [ 21 0
            ] ]
          [2015-01-14 09:18:42.944432] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /
          [2015-01-14 09:18:42.946664] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 4 ] [ 2 0 ]
            ]
          [2015-01-14 09:18:42.947323] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /mpdis
          [2015-01-14 09:18:43.955929] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 20 ] [ 21 0
            ] ]
          [2015-01-14 09:18:43.956701] E
            [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
            0-gv0-replicate-0:  metadata self heal  failed,   on /
          [2015-01-14 09:18:43.958874] E
            [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
            0-gv0-replicate-0: Unable to self-heal contents of '/mpdis'
            (possible split-brain). Please delete the file from all but
            the preferred subvolume.- Pending matrix:  [ [ 0 4 ] [ 2 0 ]
            ]
           
          Besides, I find  Input/output error shown
            below  when listing the files of under mount point:
           
          [root@dmf-wpst-2 mpdis]# ll
          total 0
          -rwxr-xr-x. 1 apache apache 0 Jan
            14 04:21 test.rep.00.00.00.00.dmf1
          -rw-r--r--. 1 apache apache 0 Jan
            14 04:21 test.rep.00.00.00.00.dmf2
           
          [root@dmf-wpst-2 mpdis]# ll
          total 0
          -rwxr-xr-x. 1 apache apache 0 Jan
            14 04:21 test.rep.00.00.00.00.dmf1
          -rw-r--r--. 1 apache apache 0 Jan
            14 04:21 test.rep.00.00.00.00.dmf2
           
          [root@dmf-wpst-2 mpdis]# ll
          ls: cannot open directory .:
            Input/output error
           
          [root@dmf-wpst-2 mpdis]# ll
          ls: cannot access
              test.rep.00.00.00.00.dmf1: Input/output error
          ls: cannot access
              test.rep.00.00.00.00.dmf2: Input/output error
          total 0
          ?????????? ? ? ? ?            ?
            test.rep.00.00.00.00.dmf1
          ?????????? ? ? ? ?            ?
            test.rep.00.00.00.00.dmf2
              Any
                tips about debugging further or getting this fixed up
                would be appreciated.  
           
          [version]:
              3.5.3
          [environment]: 
            two virtual server each has one brick :
           
          root@dmf-wpst-2 mpdis]# gluster
            volume status
          Status of volume: gv0
          Gluster
            process                                                                                               
            Port       Online   Pid
          ------------------------------------------------------------------------------
          Brick
            dmf-ha-1-glusterfs:/export/vdb1/brick                     
            49152    Y              332
          Brick
            dmf-ha-2-glusterfs:/export/vdb1/brick                     
            49154    Y              19396
          Self-heal Daemon on
            localhost                                                 
            N/A        Y              19410
          Self-heal Daemon on
            10.175.123.246                                      
            N/A        Y              999
           
          [root@dmf-wpst-1 mpdis]# gluster
            volume info
          Volume Name: gv0
          Type: Replicate
          Volume ID:
            51de44c3-f01e-486d-a6b7-10c7b7a270d7
          Status: Started
          Number of Bricks: 1 x 2 = 2
          Transport-type: tcp
          Bricks:
          Brick1:
            dmf-ha-1-glusterfs:/export/vdb1/brick
          Brick2:
            dmf-ha-2-glusterfs:/export/vdb1/brick
          Options Reconfigured:
          nfs.disable: ON
          network.ping-timeout: 2
          storage.bd-aio: on
          storage.linux-aio: on
          cluster.eager-lock: on
          performance.client-io-threads: on
          performance.cache-refresh-timeout:
            60
          performance.io-thread-count: 64
          performance.cache-size: 8GB
          cluster.server-quorum-type: none
           
            [mount-point info]:
          
            1.      
              mount command
          glusterfs
            -p /var/run/glusterfs.pid
            --volfile-server=dmf-ha-1-glusterfs
            --volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0
            /dmfcontents
           
           
            2.      
              mount point directory
            hierarchy                   
          [root@dmf-wpst-2
            /]# ls -ld /dmfcontents/
          drwxr-xr-x. 5
            root root 71 Jan 14 04:39 /dmfcontents/
                         [root@dmf-wpst-2 /]# ls
            -ld /dmfcontents/mpdis/
                        drwxr-xr-x. 2 apache apache
            89 Jan 14 04:39 /dmfcontents/mpdis/
        
        hi Jifeng Li,

                 Sorry for the delay in response. Could you post the
            output of:

            'getfattr -d -m. -e hex <brick-path>'

            'getfattr -d -m. -e hex <brick-path>/mpdis'

            'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf1'

            'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf2'

            
            Pranith

            
        _______________________________________________
        Gluster-users mailing list
        Gluster-users@xxxxxxxxxxx
        http://www.gluster.org/mailman/listinfo/gluster-users
         
      
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users