Re: memory leaks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I just wanted to confirm that, yes, this does indeed fix the problem.

Many Thanks!

Brent

On Sun, 18 Mar 2007, Anand Avati wrote:

Brent,
 The 'leak' has been plugged in the read-ahead translator of the
latest tla patch. the problem was an integer overflow issue where the
'size' parameter for the flush function was 32bit but when the file
was bigger than 4GB, the code tried to flush a region from 0 to >4GB.
changing the function prototype to have a 64bit size fixed it.

The 'leak' was actually not a 'leak' because when the file was closed,
everything was flushed and free'd.

thanks!
avati


On Mon, Mar 12, 2007 at 02:15:31PM -0400, Brent A Nelson wrote:
Attached are my present spec files for both servers (which are also my
clients); both machines use the same files.

To trigger the memory leak, all I need to do is a "dd if=/phys/blah0
of=/dev/null bs=1M", where /phys/blah0 was created with "dd if=/dev/zero
of=/phys/blah0 bs=10M count=1024".  It seems fine for a moment, but then
glusterfs starts growing and growing until it can't claim any more RAM
and becomes useless (can't ls the mountpoint, dd doesn't complete)...

Both machines are running Ubuntu Edgy + Fuse 2.6.3.

Thanks,

Brent

On Fri, 9 Mar 2007, Anand Avati wrote:

Brent,
we are unable to get the memory leak in read-ahead (we found one and
fixed, but your mail indicateds there is another). can you please mail
your exact volume spec files and the command line to initiate the I/O
which reproduces the leak?

regards,
avati


On Fri, Mar 09, 2007 at 07:25:03PM -0500, Brent A Nelson wrote:
I just confirmed it on client-side, as well.  glusterfs was stable for a
minute, and then it suddenly started increasing in size very rapidly.

Thanks,

Brent



volume disk0
  type storage/posix                   # POSIX FS translator
  option directory /share0             # Export this directory
end-volume
volume disk1
  type storage/posix                   # POSIX FS translator
  option directory /share1             # Export this directory
end-volume
volume disk2
  type storage/posix                   # POSIX FS translator
  option directory /share2             # Export this directory
end-volume
volume disk3
  type storage/posix                   # POSIX FS translator
  option directory /share3             # Export this directory
end-volume
volume disk4
  type storage/posix                   # POSIX FS translator
  option directory /share4             # Export this directory
end-volume
volume disk5
  type storage/posix                   # POSIX FS translator
  option directory /share5             # Export this directory
end-volume
volume disk6
  type storage/posix                   # POSIX FS translator
  option directory /share6             # Export this directory
end-volume
volume disk7
  type storage/posix                   # POSIX FS translator
  option directory /share7             # Export this directory
end-volume
volume disk8
  type storage/posix                   # POSIX FS translator
  option directory /share8             # Export this directory
end-volume
volume disk9
  type storage/posix                   # POSIX FS translator
  option directory /share9             # Export this directory
end-volume
volume disk10
  type storage/posix                   # POSIX FS translator
  option directory /share10            # Export this directory
end-volume
volume disk11
  type storage/posix                   # POSIX FS translator
  option directory /share11            # Export this directory
end-volume
volume disk12
  type storage/posix                   # POSIX FS translator
  option directory /share12            # Export this directory
end-volume
volume disk13
  type storage/posix                   # POSIX FS translator
  option directory /share13            # Export this directory
end-volume
volume disk14
  type storage/posix                   # POSIX FS translator
  option directory /share14            # Export this directory
end-volume
volume disk15
  type storage/posix                   # POSIX FS translator
  option directory /share15            # Export this directory
end-volume

volume share0
  type performance/io-threads
  option thread-count 8
  subvolumes disk0
end-volume

volume share1
  type performance/io-threads
  option thread-count 8
  subvolumes disk1
end-volume

volume share2
  type performance/io-threads
  option thread-count 8
  subvolumes disk2
end-volume

volume share3
  type performance/io-threads
  option thread-count 8
  subvolumes disk3
end-volume

volume share4
  type performance/io-threads
  option thread-count 8
  subvolumes disk4
end-volume

volume share5
  type performance/io-threads
  option thread-count 8
  subvolumes disk5
end-volume

volume share6
  type performance/io-threads
  option thread-count 8
  subvolumes disk6
end-volume

volume share7
  type performance/io-threads
  option thread-count 8
  subvolumes disk7
end-volume

volume share8
  type performance/io-threads
  option thread-count 8
  subvolumes disk8
end-volume

volume share9
  type performance/io-threads
  option thread-count 8
  subvolumes disk9
end-volume

volume share10
  type performance/io-threads
  option thread-count 8
  subvolumes disk10
end-volume

volume share11
  type performance/io-threads
  option thread-count 8
  subvolumes disk11
end-volume

volume share12
  type performance/io-threads
  option thread-count 8
  subvolumes disk12
end-volume

volume share13
  type performance/io-threads
  option thread-count 8
  subvolumes disk13
end-volume

volume share14
  type performance/io-threads
  option thread-count 8
  subvolumes disk14
end-volume

volume share15
  type performance/io-threads
  option thread-count 8
  subvolumes disk15
end-volume

volume server
  type protocol/server
  option transport-type tcp/server     # For TCP/IP transport
# option transport-type ib-sdp/server  # For Infiniband transport
# option bind-address 192.168.1.10     # Default is to listen on all interfaces
# option listen-port 6996               # Default is 6996
  option client-volume-filename /etc/glusterfs/glusterfs-client.vol
  subvolumes share0 share1 share2 share3 share4 share5 share6 share7 share8 share9 share10 share11 share12 share13 share14 share15
# NOTE: Access to any volume through protocol/server is denied by
# default. You need to explicitly grant access through # "auth"
# option.
  option auth.ip.share0.allow 128.227.64.*,128.227.89.*
  option auth.ip.share1.allow 128.227.64.*,128.227.89.*
  option auth.ip.share2.allow 128.227.64.*,128.227.89.*
  option auth.ip.share3.allow 128.227.64.*,128.227.89.*
  option auth.ip.share4.allow 128.227.64.*,128.227.89.*
  option auth.ip.share5.allow 128.227.64.*,128.227.89.*
  option auth.ip.share6.allow 128.227.64.*,128.227.89.*
  option auth.ip.share7.allow 128.227.64.*,128.227.89.*
  option auth.ip.share8.allow 128.227.64.*,128.227.89.*
  option auth.ip.share9.allow 128.227.64.*,128.227.89.*
  option auth.ip.share10.allow 128.227.64.*,128.227.89.*
  option auth.ip.share11.allow 128.227.64.*,128.227.89.*
  option auth.ip.share12.allow 128.227.64.*,128.227.89.*
  option auth.ip.share13.allow 128.227.64.*,128.227.89.*
  option auth.ip.share14.allow 128.227.64.*,128.227.89.*
  option auth.ip.share15.allow 128.227.64.*,128.227.89.*
end-volume

volume share0-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share0       # name of the remote volume
end-volume
volume share0-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share0       # name of the remote volume
end-volume
volume share1-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share1       # name of the remote volume
end-volume
volume share1-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share1       # name of the remote volume
end-volume
volume share2-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share2       # name of the remote volume
end-volume
volume share2-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share2       # name of the remote volume
end-volume
volume share3-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share3       # name of the remote volume
end-volume
volume share3-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share3       # name of the remote volume
end-volume
volume share4-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share4       # name of the remote volume
end-volume
volume share4-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share4       # name of the remote volume
end-volume
volume share5-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share5       # name of the remote volume
end-volume
volume share5-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share5       # name of the remote volume
end-volume
volume share6-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share6       # name of the remote volume
end-volume
volume share6-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share6       # name of the remote volume
end-volume
volume share7-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share7       # name of the remote volume
end-volume
volume share7-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share7       # name of the remote volume
end-volume
volume share8-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share8       # name of the remote volume
end-volume
volume share8-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share8       # name of the remote volume
end-volume
volume share9-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share9       # name of the remote volume
end-volume
volume share9-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share9       # name of the remote volume
end-volume
volume share10-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share10      # name of the remote volume
end-volume
volume share10-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share10      # name of the remote volume
end-volume
volume share11-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share11      # name of the remote volume
end-volume
volume share11-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share11      # name of the remote volume
end-volume
volume share12-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share12      # name of the remote volume
end-volume
volume share12-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share12      # name of the remote volume
end-volume
volume share13-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share13      # name of the remote volume
end-volume
volume share13-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share13      # name of the remote volume
end-volume
volume share14-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share14      # name of the remote volume
end-volume
volume share14-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share14      # name of the remote volume
end-volume
volume share15-0
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.241    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share15      # name of the remote volume
end-volume
volume share15-1
  type protocol/client
  option transport-type tcp/client     # for TCP/IP transport
# option transport-type ib-sdp/client  # for Infiniband transport
  option remote-host 128.227.64.242    # IP address of the remote brick
# option remote-port 6996              # default server port is 6996
  option remote-subvolume share15      # name of the remote volume
end-volume

volume mirror0
  type cluster/afr
  subvolumes share0-0 share0-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror1
  type cluster/afr
  subvolumes share1-1 share1-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror2
  type cluster/afr
  subvolumes share2-0 share2-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror3
  type cluster/afr
  subvolumes share3-1 share3-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror4
  type cluster/afr
  subvolumes share4-0 share4-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror5
  type cluster/afr
  subvolumes share5-1 share5-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror6
  type cluster/afr
  subvolumes share6-0 share6-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror7
  type cluster/afr
  subvolumes share7-1 share7-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror8
  type cluster/afr
  subvolumes share8-0 share8-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror9
  type cluster/afr
  subvolumes share9-1 share9-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror10
  type cluster/afr
  subvolumes share10-0 share10-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror11
  type cluster/afr
  subvolumes share11-1 share11-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror12
  type cluster/afr
  subvolumes share12-0 share12-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror13
  type cluster/afr
  subvolumes share13-1 share13-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror14
  type cluster/afr
  subvolumes share14-0 share14-1
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume
volume mirror15
  type cluster/afr
  subvolumes share15-1 share15-0
  option replicate *:2     # Do not leave space before or after "," and ":"
end-volume

volume mirrors
  type cluster/unify
  subvolumes mirror0 mirror1 mirror2 mirror3 mirror4 mirror5 mirror6 mirror7 mirror8 mirror9 mirror10 mirror11 mirror12 mirror13 mirror14 mirror15
#  option scheduler rr
  option scheduler alu
  option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
  option alu.disk-usage.entry-threshold 2GB
  option alu.disk-usage.exit-threshold 60MB
  option alu.limits.min-free-disk 5GB
  option alu.stat-refresh.interval 10sec
end-volume

volume statprefetch
  type performance/stat-prefetch
  option cache-seconds 2
  subvolumes mirrors
end-volume

volume writebehind
  type performance/write-behind
  option aggregate-size 131072 # in bytes
  subvolumes statprefetch
end-volume

volume readahead
  type performance/read-ahead
  option page-size 65536 ### in bytes
  option page-count 16 ### memory cache size is page-count x page-size per file
  subvolumes writebehind
end-volume


--
Shaw's Principle:
       Build a system that even a fool can use,
       and only a fool will want to use it.





[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux