Re: Memory leak in 3.6.*?

Yannick Perret <yannick.perret@xxxxxxxxxxxxx> · Fri, 22 Jul 2016 15:31:40 +0200



    Note: I'm have a dev client machine so
      I can perform tests or recompile glusterfs client if it can helps
      getting data about that.

      
      I did not test this problem against 3.7.x version as my 2 servers
      are in use and I can't upgrade them at this time, and 3.7 clients
      are not compatible with 3.6 servers (as far as I can see from my
      tests).

      
      --

      Y.

      
      Le 22/07/2016 14:06, Yannick Perret a écrit :

    
    Hello,
      

      some times ago I posted about a memory leak in client process, but
      it was on a very old 32bit machine (both kernel and OS) and I
      don't found evidences about a similar problem on our recent
      machines.
      

      But I performed more tests and I have the same problem.
      

      Clients are 64bit Debian 8.2 machines. Glusterfs client on these
      machines is compiled from sources with activated stuff:
      

      FUSE client          : yes
      

      Infiniband verbs     : no
      

      epoll IO multiplex   : yes
      

      argp-standalone      : no
      

      fusermount           : yes
      

      readline             : yes
      

      georeplication       : yes
      

      Linux-AIO            : no
      

      Enable Debug         : no
      

      systemtap            : no
      

      Block Device xlator  : no
      

      glupy                : no
      

      Use syslog           : yes
      

      XML output           : yes
      

      QEMU Block formats   : no
      

      Encryption xlator    : yes
      

      Erasure Code xlator  : yes
      

      I tested both 3.6.7 and 3.6.9 version on client (3.6.7 is the one
      installed on our machines, even on servers, 3.6.9 is for testing
      with last 3.6 version).
      

      Here are the operations on the client (also performed with similar
      results with 3.6.7 version):
      

      # /usr/local/sbin/glusterfs --version
      

      glusterfs 3.6.9 built on Jul 22 2016 13:27:42
      

      (…)
      

      # mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
      

      # cd /usr/
      

      # cp -Rp * /zog/TEMP/
      

      Then monitoring memory used by glusterfs process while 'cp' is
      running (resp. VSZ and RSS from 'ps'):
      

      284740 70232
      

      284740 70232
      

      284876 71704
      

      285000 72684
      

      285136 74008
      

      285416 75940
      

      (…)
      

      368684 151980
      

      369324 153768
      

      369836 155576
      

      370092 156192
      

      370092 156192
      

      Here both sizes are stable and correspond to the end of 'cp'
      command.
      

      If I restart an other 'cp' (even on the same directories) size
      starts again to increase.
      

      If I perform a 'ls -lR' in the directory size also increase:
      

      370756 192488
      

      389964 212148
      

      390948 213232
      

      (here I ^C the 'ls')
      

      When doing nothing the size don't increase but never decrease
      (calling 'sync' don't change the situation).
      

      Sending a HUP signal to glusterfs process also increases memory
      (390948 213324 → 456484 213320).
      

      Changing volume configuration (changing
      diagnostics.client-sys-log-level value) don't change anything.
      

      Here the actual ps:
      

      root     17041  4.9  5.2 456484 213320 ?       Ssl  13:29   1:21
      /usr/local/sbin/glusterfs --volfile-server=sto1.my.domain
      --volfile-id=BACKUP-ADMIN-DATA /zog
      

      Of course umouting/remounting fall back to "start" size:
      

      # umount /zog
      

      # mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
      

      → root     28741  0.3  0.7 273320 30484 ?        Ssl  13:57   0:00
      /usr/local/sbin/glusterfs --volfile-server=sto1.my.domain
      --volfile-id=BACKUP-ADMIN-DATA /zog
      

      I didn't saw this before because most of our volumes are mounted
      "on demand" for some storage activities or are permanently mounted
      but with very few activity.
      

      But clearly this memory usage driff is a long-term problem. On the
      old 32bit machine I had this problem ("solved" by using NFS mounts
      in order to wait for this old machine to be replaced) and it lead
      to glusterfs being killed by OS when out of free memory. It was
      faster than what I describe here but it's just a question of time.
      

      Thanks for any help about that.
      

      Regards,
      

      --
      

      Y.
      

      The corresponding volume on servers is (if it can help):
      

      Volume Name: BACKUP-ADMIN-DATA
      

      Type: Replicate
      

      Volume ID: 306d57f3-fb30-4bcc-8687-08bf0a3d7878
      

      Status: Started
      

      Number of Bricks: 1 x 2 = 2
      

      Transport-type: tcp
      

      Bricks:
      

      Brick1: sto1.my.domain:/glusterfs/backup-admin/data
      

      Brick2: sto2.my.domain:/glusterfs/backup-admin/data
      

      Options Reconfigured:
      

      diagnostics.client-sys-log-level: WARNING
      

      _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
    
    
Attachment:
smime.p7s

Description: Signature cryptographique S/MIME
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users