I also neglected to mention that the underlying filesystem is ext3. On 3/24/2010 3:44 AM, Jeremy Enos wrote: > I haven't tried all performance options disabled yet- I can try that > tomorrow when the resource frees up. I was actually asking first > before blindly trying different configuration matrices in case there's > a clear direction I should take with it. I'll let you know. > > Jeremy > > On 3/24/2010 2:54 AM, Stephan von Krawczynski wrote: >> Hi Jeremy, >> >> have you tried to reproduce with all performance options disabled? >> They are >> possibly no good idea on a local system. >> What local fs do you use? >> >> >> -- >> Regards, >> Stephan >> >> >> On Tue, 23 Mar 2010 19:11:28 -0500 >> Jeremy Enos<jenos at ncsa.uiuc.edu> wrote: >> >>> Stephan is correct- I primarily did this test to show a demonstrable >>> overhead example that I'm trying to eliminate. It's pronounced enough >>> that it can be seen on a single disk / single node configuration, which >>> is good in a way (so anyone can easily repro). >>> >>> My distributed/clustered solution would be ideal if it were fast enough >>> for small block i/o as well as large block- I was hoping that single >>> node systems would achieve that, hence the single node test. Because >>> the single node test performed poorly, I eventually reduced down to >>> single disk to see if it could still be seen, and it clearly can be. >>> Perhaps it's something in my configuration? I've pasted my config >>> files >>> below. >>> thx- >>> >>> Jeremy >>> >>> ######################glusterfsd.vol###################### >>> volume posix >>> type storage/posix >>> option directory /export >>> end-volume >>> >>> volume locks >>> type features/locks >>> subvolumes posix >>> end-volume >>> >>> volume disk >>> type performance/io-threads >>> option thread-count 4 >>> subvolumes locks >>> end-volume >>> >>> volume server-ib >>> type protocol/server >>> option transport-type ib-verbs/server >>> option auth.addr.disk.allow * >>> subvolumes disk >>> end-volume >>> >>> volume server-tcp >>> type protocol/server >>> option transport-type tcp/server >>> option auth.addr.disk.allow * >>> subvolumes disk >>> end-volume >>> >>> ######################ghome.vol###################### >>> >>> #-----------IB remotes------------------ >>> volume ghome >>> type protocol/client >>> option transport-type ib-verbs/client >>> # option transport-type tcp/client >>> option remote-host acfs >>> option remote-subvolume raid >>> end-volume >>> >>> #------------Performance Options------------------- >>> >>> volume readahead >>> type performance/read-ahead >>> option page-count 4 # 2 is default option >>> option force-atime-update off # default is off >>> subvolumes ghome >>> end-volume >>> >>> volume writebehind >>> type performance/write-behind >>> option cache-size 1MB >>> subvolumes readahead >>> end-volume >>> >>> volume cache >>> type performance/io-cache >>> option cache-size 1GB >>> subvolumes writebehind >>> end-volume >>> >>> ######################END###################### >>> >>> >>> >>> On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote: >>>> On Tue, 23 Mar 2010 02:59:35 -0600 (CST) >>>> "Tejas N. Bhise"<tejas at gluster.com> wrote: >>>> >>>> >>>>> Out of curiosity, if you want to do stuff only on one machine, >>>>> why do you want to use a distributed, multi node, clustered, >>>>> file system ? >>>>> >>>> Because what he does is a very good way to show the overhead >>>> produced only by >>>> glusterfs and nothing else (i.e. no network involved). >>>> A pretty relevant test scenario I would say. >>>> >>>> -- >>>> Regards, >>>> Stephan >>>> >>>> >>>> >>>>> Am I missing something here ? >>>>> >>>>> Regards, >>>>> Tejas. >>>>> >>>>> ----- Original Message ----- >>>>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu> >>>>> To: gluster-users at gluster.org >>>>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, >>>>> Kolkata, Mumbai, New Delhi >>>>> Subject: gluster local vs local = gluster x4 slower >>>>> >>>>> This test is pretty easy to replicate anywhere- only takes 1 disk, >>>>> one >>>>> machine, one tarball. Untarring to local disk directly vs thru >>>>> gluster >>>>> is about 4.5x faster. At first I thought this may be due to a >>>>> slow host >>>>> (Opteron 2.4ghz). But it's not- same configuration, on a much faster >>>>> machine (dual 3.33ghz Xeon) yields the performance below. >>>>> >>>>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER#### >>>>> [root at ac33 jenos]# time tar xzf >>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz >>>>> >>>>> real 0m41.290s >>>>> user 0m14.246s >>>>> sys 0m2.957s >>>>> >>>>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)#### >>>>> [root at ac33 jenos]# cd /export/jenos/ >>>>> [root at ac33 jenos]# time tar xzf >>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz >>>>> >>>>> real 0m8.983s >>>>> user 0m6.857s >>>>> sys 0m1.844s >>>>> >>>>> ####THESE ARE TEST FILE DETAILS#### >>>>> [root at ac33 jenos]# tar tzvf >>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l >>>>> 109 >>>>> [root at ac33 jenos]# ls -l >>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz >>>>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32 >>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz >>>>> [root at ac33 jenos]# >>>>> >>>>> These are the relevant performance options I'm using in my .vol file: >>>>> >>>>> #------------Performance Options------------------- >>>>> >>>>> volume readahead >>>>> type performance/read-ahead >>>>> option page-count 4 # 2 is default option >>>>> option force-atime-update off # default is off >>>>> subvolumes ghome >>>>> end-volume >>>>> >>>>> volume writebehind >>>>> type performance/write-behind >>>>> option cache-size 1MB >>>>> subvolumes readahead >>>>> end-volume >>>>> >>>>> volume cache >>>>> type performance/io-cache >>>>> option cache-size 1GB >>>>> subvolumes writebehind >>>>> end-volume >>>>> >>>>> What can I do to improve gluster's performance? >>>>> >>>>> Jeremy >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> >>>>> >>>> >>
