Re: [Gluster-devel] heal hanging

Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx> · Thu, 21 Jan 2016 21:40:25 +0530



    On 01/21/2016 09:26 PM, Glomski,
      Patrick wrote:

    
        I should mention that the problem is not currently
          occurring and there are no heals (output appended). By
          restarting the gluster services, we can stop the crawl, which
          lowers the load for a while. Subsequent crawls seem to finish
          properly. For what it's worth, files/folders that show up in
          the 'volume info' output during a hung crawl don't seem to be
          anything out of the ordinary. 

          
          Over the past four days, the typical time before the problem
          recurs after suppressing it in this manner is an hour. Last
          night when we reached out to you was the last time it happened
          and the load has been low since (a relief).  David believes
          that recursively listing the files (ls -alR or similar) from a
          client mount can force the issue to happen, but obviously I'd
          rather not unless we have some precise thing we're looking
          for. Let me know if you'd like me to attempt to drive the
          system unstable like that and what I should look for. As it's
          a production system, I'd rather not leave it in this state for
          long.

        
    Will it be possible to send glustershd, mount logs of the past 4
    days? I would like to see if this is because of directory self-heal
    going wild (Ravi is working on throttling feature for 3.8, which
    will allow to put breaks on self-heal traffic)

    
    Pranith

    
        [root@gfs01a xattrop]# gluster volume heal homegfs info

          Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/

          Number of entries: 0

          
          Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/

          Number of entries: 0

          
          Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/

          Number of entries: 0

          
          Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/

          Number of entries: 0

          
          Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/

          Number of entries: 0

          
          Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/

          Number of entries: 0

          
          Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/

          Number of entries: 0

          
          Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/

          Number of entries: 0

          
        On Thu, Jan 21, 2016 at 10:40 AM,
          Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx> wrote:

          
                On 01/21/2016 08:25 PM, Glomski, Patrick wrote:

                
                    Hello, Pranith. The typical behavior is that
                      the %cpu on a glusterfsd process jumps to number
                      of processor cores available (800% or 1200%,
                      depending on the pair of nodes involved) and the
                      load average on the machine goes very high (~20).
                      The volume's heal statistics output shows that it
                      is crawling one of the bricks and trying to heal,
                      but this crawl hangs and never seems to finish.

                    
                    The number of files in the xattrop directory varies
                    over time, so I ran a wc -l as you requested
                    periodically for some time and then started
                    including a datestamped list of the files that were
                    in the xattrops directory on each brick to see which
                    were persistent. All bricks had files in the xattrop
                    folder, so all results are attached.

                  
               Thanks this info is helpful. I don't see a lot of
              files. Could you give output of "gluster volume heal
              <volname> info"? Is there any directory in there
              which is LARGE?

                  
                  Pranith
              
                
                      Please let me know if there is anything else
                        I can provide.

                      
                      Patrick

                      
                      On Thu, Jan 21, 2016 at
                        12:01 AM, Pranith Kumar Karampuri <pkarampu@xxxxxxxxxx>
                        wrote:

                        
                           hey,

                                   Which process is consuming so much
                            cpu? I went through the logs you gave me. I
                            see that the following files are in gfid
                            mismatch state:

                            
<066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,

<1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,

<ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,

                            
                            Could you give me the output of "ls
                            <brick-path>/indices/xattrop | wc -l"
                            output on all the bricks which are acting
                            this way? This will tell us the number of
                            pending self-heals on the system.

                            
                            Pranith
                            
                              
                                On 01/20/2016 09:26 PM, David
                                  Robinson wrote:

                                
                                  resending with parsed logs... 
                                   
                                  
                                          I am having issues with
                                            3.6.6 where the load will
                                            spike up to 800% for one of
                                            the glusterfsd processes and
                                            the users can no longer
                                            access the system.  If I
                                            reboot the node, the heal
                                            will finish normally after a
                                            few minutes and the system
                                            will be responsive, but a
                                            few hours later the issue
                                            will start again.  It look
                                            like it is hanging in a heal
                                            and spinning up the load on
                                            one of the bricks.  The heal
                                            gets stuck and says it is
                                            crawling and never returns. 
                                            After a few minutes of the
                                            heal saying it is crawling,
                                            the load spikes up and the
                                            mounts become unresponsive.
                                           
                                          Any suggestions on how to
                                            fix this?  It has us stopped
                                            cold as the user can no
                                            longer access the systems
                                            when the load spikes... Logs
                                            attached.
                                           
                                          System setup info is: 
                                           
                                          [root@gfs01a ~]# gluster
                                            volume info homegfs

                                             
                                            Volume Name: homegfs

                                            Type: Distributed-Replicate

                                            Volume ID:
                                            1e32672a-f1b7-4b58-ba94-58c085e59071

                                            Status: Started

                                            Number of Bricks: 4 x 2 = 8

                                            Transport-type: tcp

                                            Bricks:

                                            Brick1:
                                            gfsib01a.corvidtec.com:/data/brick01a/homegfs

                                            Brick2:
                                            gfsib01b.corvidtec.com:/data/brick01b/homegfs

                                            Brick3:
                                            gfsib01a.corvidtec.com:/data/brick02a/homegfs

                                            Brick4:
                                            gfsib01b.corvidtec.com:/data/brick02b/homegfs

                                            Brick5:
                                            gfsib02a.corvidtec.com:/data/brick01a/homegfs

                                            Brick6:
                                            gfsib02b.corvidtec.com:/data/brick01b/homegfs

                                            Brick7:
                                            gfsib02a.corvidtec.com:/data/brick02a/homegfs

                                            Brick8:
                                            gfsib02b.corvidtec.com:/data/brick02b/homegfs

                                            Options Reconfigured:

                                            performance.io-thread-count:
                                            32

                                            performance.cache-size:
                                            128MB

                                            performance.write-behind-window-size:

                                            128MB

                                            server.allow-insecure: on

                                            network.ping-timeout: 42

                                            storage.owner-gid: 100

                                            geo-replication.indexing:
                                            off

                                            geo-replication.ignore-pid-check:
                                            on

                                            changelog.changelog: off

                                            changelog.fsync-interval: 3

                                            changelog.rollover-time: 15

                                            server.manage-gids: on

                                            diagnostics.client-log-level:
                                            WARNING
                                           
                                          [root@gfs01a ~]# rpm -qa
                                            | grep gluster

gluster-nagios-common-0.1.1-0.el6.noarch

glusterfs-fuse-3.6.6-1.el6.x86_64

glusterfs-debuginfo-3.6.6-1.el6.x86_64

glusterfs-libs-3.6.6-1.el6.x86_64

glusterfs-geo-replication-3.6.6-1.el6.x86_64

glusterfs-api-3.6.6-1.el6.x86_64

glusterfs-devel-3.6.6-1.el6.x86_64

glusterfs-api-devel-3.6.6-1.el6.x86_64

                                            glusterfs-3.6.6-1.el6.x86_64

glusterfs-cli-3.6.6-1.el6.x86_64

glusterfs-rdma-3.6.6-1.el6.x86_64

samba-vfs-glusterfs-4.1.11-2.el6.x86_64

glusterfs-server-3.6.6-1.el6.x86_64

glusterfs-extra-xlators-3.6.6-1.el6.x86_64

                                          
                              _______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
                            
                            
_______________________________________________

                          Gluster-users mailing list

                          Gluster-users@xxxxxxxxxxx

                          http://www.gluster.org/mailman/listinfo/gluster-users

                        
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users