Tejas,
We still have hundreds of GBs to copy, and have not put the new file
system into the test. So far the clients works all fine. I mean the
commands like ls, mkdir, touch, and etc.
Thanks again for your time.
regards,
Chris
On Wed, 2010-04-14 at 23:04 -0600, Tejas N. Bhise wrote:
> Chris,
>
> By the way, after the copy is done, how is the system responding to
> regular access ? In the sense, was the problem with copy also
> carried forward as more trouble seen with subsequent access of
> data over glusterfs ?
>
> Regards,
> Tejas.
>
> ----- Original Message -----
> From: "Chris Jin" <chris at pikicentral.com>
> To: "Tejas N. Bhise" <tejas at gluster.com>
> Cc: "gluster-users" <gluster-users at gluster.org>
> Sent: Thursday, April 15, 2010 9:48:42 AM
> Subject: Re: Memory usage high on server sides
>
> Hi Tejas,
>
> > Problems you saw -
> >
> > 1) High memory usage on client where gluster volume is mounted
>
> Memory usage for clients is 0% after copying.
> $ps auxf
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 19692 1.3 0.0 262148 6980 ? Ssl Apr12
> 61:33 /sbin/glusterfs --log-level=NORMAL
> --volfile=/u2/git/modules/shared/glusterfs/clients/r2/c2.vol /gfs/r2/f2
>
> > 2) High memory usage on server
> Yes.
> $ps auxf
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 26472 2.2 29.1 718100 600260 ? Ssl Apr09 184:09
> glusterfsd -f /etc/glusterfs/servers/r2/f1.vol
> root 26485 1.8 39.8 887744 821384 ? Ssl Apr09 157:16
> glusterfsd -f /etc/glusterfs/servers/r2/f2.vol
>
> > 3) 2 days to copy 300 GB data
> More than 700GB. There are two folders. The first one is copied to
> server 1 and server 2, and the second one is copied to server 2 and
> server 3. The vol files are below.
>
> > About the config, can you provide the following for both old and new systems -
> >
> > 1) OS and kernel level on gluster servers and clients
> Debian Kernel 2.6.18-6-amd64
>
> $uname -a
> Linux fs2 2.6.18-6-amd64 #1 SMP Tue Aug 19 04:30:56 UTC 2008 x86_64
> GNU/Linux
>
> > 2) volume file from servers and clients
>
> #####Server Vol file (f1.vol)
> # The same settings for f2.vol and f3.vol, just different dirs and ports
> # f1 f3 for Server 1, f1 f2 for Server 2, f2 f3 for Server 3
> volume posix1
> type storage/posix
> option directory /gfs/r2/f1
> end-volume
>
> volume locks1
> type features/locks
> subvolumes posix1
> end-volume
>
> volume brick1
> type performance/io-threads
> option thread-count 8
> subvolumes locks1
> end-volume
>
> volume server-tcp
> type protocol/server
> option transport-type tcp
> option auth.addr.brick1.allow 192.168.0.*
> option transport.socket.listen-port 6991
> option transport.socket.nodelay on
> subvolumes brick1
> end-volume
>
> #####Client Vol file (c1.vol)
> # The same settings for c2.vol and c3.vol
> # s2 s3 for c2, s3 s1 for c3
> volume s1
> type protocol/client
> option transport-type tcp
> option remote-host 192.168.0.31
> option transport.socket.nodelay on
> option transport.remote-port 6991
> option remote-subvolume brick1
> end-volume
>
> volume s2
> type protocol/client
> option transport-type tcp
> option remote-host 192.168.0.32
> option transport.socket.nodelay on
> option transport.remote-port 6991
> option remote-subvolume brick1
> end-volume
>
> volume mirror
> type cluster/replicate
> option data-self-heal off
> option metadata-self-heal off
> option entry-self-heal off
> subvolumes s1 s2
> end-volume
>
> volume writebehind
> type performance/write-behind
> option cache-size 100MB
> option flush-behind off
> subvolumes mirror
> end-volume
>
> volume iocache
> type performance/io-cache
> option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 *
> 0.2 / 1024}' | cut -f1 -d.`MB
> option cache-timeout 1
> subvolumes writebehind
> end-volume
>
> volume quickread
> type performance/quick-read
> option cache-timeout 1
> option max-file-size 256Kb
> subvolumes iocache
> end-volume
>
> volume statprefetch
> type performance/stat-prefetch
> subvolumes quickread
> end-volume
>
>
> > 3) Filesystem type of backend gluster subvolumes
> ext3
>
> > 4) How close to full the backend subvolumes are
> New 2T hard disks for each server.
>
> > 5) The exact copy command .. did you mount the volumes from
> > old and new system on a single machine and did cp or used rsync
> > or some other method ? If something more than just a cp, please
> > send the exact command line you used.
> The old file system uses DRBD and NFS.
> The exact command is
> sudo cp -R -v -p -P /nfsmounts/nfs3/photo .
>
> > 6) How many files/directories ( tentative ) in that 300GB data ( would help in
> > trying to reproduce inhouse with a smaller test bed ).
> I cannot tell, but the file sizes are between 1KB to 200KB, average
> around 20KB.
>
> > 7) Was there other load on the new or old system ?
> The old systems are still used for web servers.
> The new systems are on the same servers but different hard disks.
>
> > 8) Any other patterns you noticed.
> There is once that one client tried to connect one server with external
> IP address.
> Using distribute translator across all three mirrors will make system
> twice slower than using three mounted folders.
>
> Is this information enough?
>
> Please take a look.
>
> Regards,
>
> Chris
>
>
>
