Bellow. I've found that I get a performance hit if I add read cache or whitebehind. Server conf: ##Open vols volume posix1 type storage/posix option directory /gluster/export1 end-volume volume posix2 type storage/posix option directory /gluster/export2 end-volume volume posix3 type storage/posix option directory /gluster/export3 end-volume volume posix4 type storage/posix option directory /gluster/export4 end-volume volume posix5 type storage/posix option directory /gluster/export5 end-volume volume posix6 type storage/posix option directory /gluster/export6 end-volume ## Add the ability to lock files etc volume locks1 type features/locks subvolumes posix1 end-volume volume locks2 type features/locks subvolumes posix2 end-volume volume locks3 type features/locks subvolumes posix3 end-volume volume locks4 type features/locks subvolumes posix4 end-volume volume locks5 type features/locks subvolumes posix5 end-volume volume locks6 type features/locks subvolumes posix6 end-volume ## Preformance translators volume brick1 type performance/io-threads option thread-count 8 subvolumes locks1 end-volume volume brick2 type performance/io-threads option thread-count 8 subvolumes locks2 end-volume volume brick3 type performance/io-threads option thread-count 8 subvolumes locks3 end-volume volume brick4 type performance/io-threads option thread-count 8 subvolumes locks4 end-volume volume brick5 type performance/io-threads option thread-count 8 subvolumes locks5 end-volume volume brick6 type performance/io-threads option thread-count 8 subvolumes locks6 end-volume ##export the lot #volume brick6 # type debug/trace # subvolumes trace_brick6 #end-volume volume server type protocol/server option transport-type tcp/server option auth.addr.brick1.allow * option auth.addr.brick2.allow * option auth.addr.brick3.allow * option auth.addr.brick4.allow * option auth.addr.brick5.allow * option auth.addr.brick6.allow * subvolumes brick1 brick2 brick3 brick4 brick5 brick6 end-volume Vol file: ##Clent config ##import all the briks from all the mirrors and mirror them volume glust1a_1 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick1 end-volume volume glust1b_1 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick1 end-volume volume mirror1_1 type cluster/replicate subvolumes glust1a_1 glust1b_1 end-volume volume glust1a_2 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick2 end-volume volume glust1b_2 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick2 end-volume volume mirror1_2 type cluster/replicate subvolumes glust1a_2 glust1b_2 end-volume volume glust1a_3 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick3 end-volume volume glust1b_3 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick3 end-volume volume mirror1_3 type cluster/replicate subvolumes glust1a_3 glust1b_3 end-volume volume glust1a_4 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick4 end-volume volume glust1b_4 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick4 end-volume volume mirror1_4 type cluster/replicate subvolumes glust1a_4 glust1b_4 end-volume volume glust1a_5 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick5 end-volume volume glust1b_5 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick5 end-volume volume mirror1_5 type cluster/replicate subvolumes glust1a_5 glust1b_5 end-volume volume glust1a_6 type protocol/client option transport-type tcp/client option remote-host glust1a option ping-timeout 30 option remote-subvolume brick6 end-volume volume glust1b_6 type protocol/client option transport-type tcp/client option remote-host glust1b option ping-timeout 30 option remote-subvolume brick6 end-volume volume mirror1_6 type cluster/replicate subvolumes glust1a_6 glust1b_6 end-volume ##place it all into a hash volume dist type cluster/distribute subvolumes mirror1_1 mirror1_2 mirror1_3 mirror1_4 mirror1_5 mirror1_6 end-volume > -----Original Message----- > From: Pavan Vilas Sondur [mailto:pavan at gluster.com] > Sent: 23 September 2009 13:38 > To: Hiren Joshi > Cc: gluster-users at gluster.org > Subject: Re: Rsync > > Can you send us the volfiles and logfiles as well? > > Pavan > > On 23/09/09 10:08 +0100, Hiren Joshi wrote: > > 2.0.6-1.el5, it's a 6 brick distributed system which is > mirrored over 2 > > machines (each brick has a mirror on another machine, these > mirrors are > > put in the hash). > > > > An update, after running the rsync for a day, I killed it > and remounted > > all the disks (the underlying filesystem, not the gluster) > with noatime, > > the rsync completed in about 600 minutes. I'm now going to > try one level > > up (about 1,000,000,000 dirs). > > > > > -----Original Message----- > > > From: Pavan Vilas Sondur [mailto:pavan at gluster.com] > > > Sent: 23 September 2009 07:55 > > > To: Hiren Joshi > > > Cc: gluster-users at gluster.org > > > Subject: Re: Rsync > > > > > > Hi Hiren, > > > What glusterfs version are you using? Can you send us the > > > volfiles and the log files. > > > > > > Pavan > > > > > > On 22/09/09 16:01 +0100, Hiren Joshi wrote: > > > > I forgot to mention, the mount is mounted with > direct-io, would this > > > > make a difference? > > > > > > > > > -----Original Message----- > > > > > From: gluster-users-bounces at gluster.org > > > > > [mailto:gluster-users-bounces at gluster.org] On Behalf Of > > > Hiren Joshi > > > > > Sent: 22 September 2009 11:40 > > > > > To: gluster-users at gluster.org > > > > > Subject: Rsync > > > > > > > > > > Hello all, > > > > > > > > > > I'm getting what I think is bizarre behaviour.... I have > > > about 400G to > > > > > rsync (rsync -av) onto a gluster share, the data is > in a directory > > > > > structure which has about 1000 directories per parent and > > > about 1000 > > > > > directories in each of them. > > > > > > > > > > When I try to rsync an end leaf directory (this has about 4 > > > > > dirs and 100 > > > > > files in each) the operation takes about 10 seconds. When I > > > > > go one level > > > > > above (1000 dirs with about 4 dirs in each with about 100 > > > > > files in each) > > > > > the operation takes about 10 minutes. > > > > > > > > > > Now, if I then go one level above that (that's 1000 dirs with > > > > > 1000 dirs > > > > > in each with about 4 dirs in each with about 100 files in > > > each) the > > > > > operation takes days! Top shows glusterfsd takes 300-600% > > > cpu usage > > > > > (2X4core), I have about 48G of memory (usage is 0% as > expected). > > > > > > > > > > Has anyone seen anything like this? How can I speed it up? > > > > > > > > > > Thanks, > > > > > > > > > > Josh. > > > > > > > > > _______________________________________________ > > > > Gluster-users mailing list > > > > Gluster-users at gluster.org > > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > >
