The behavior is still the same [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:33-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 264373 (258K) [text/html] Saving to: `/dev/null' 100%[====================================================================================================================>] 264,373 --.-K/s in 0.003s 12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373] [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:35-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 0 [text/html] Saving to: `/dev/null' [ <=> ] 0 --.-K/s in 0s 12:20:35 (0.00 B/s) - `/dev/null' saved [0/0] [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:38-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 0 [text/html] Saving to: `/dev/null' [ <=> ] 0 --.-K/s in 0s 12:20:38 (0.00 B/s) - `/dev/null' saved [0/0] [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" http://c0-3/cafe/siam/topic/F7800428/F7800428.html --12:20:39-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html Resolving c0-3... 10.1.255.251 Connecting to c0-3|10.1.255.251|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 264373 (258K) [text/html] Saving to: `/dev/null' 100%[====================================================================================================================>] 264,373 --.-K/s in 0.003s 12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373] [root at compute-0-0 ~]# 4 consequtive wget yield different content length. Note that, this only happened with worker + booster, prefork has no problem. And the web server seems to use correct version of glusterfs [root at compute-0-3 ~]# glusterfs -V glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 Repository revision: v2.0.5-13-g5e3ca25 Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root at compute-0-3 ~]# And all glusterfsd too [root at fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6 compute-0-6: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-8: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-9: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 compute-0-7: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 [root at fe1 ~]# Don't sure if there is anyway to query version from running glusterfsd, but I did /etc/init.d/glusterfsd restart already. 2009/8/3 Shehjar Tikoo <shehjart at gluster.com> > Somsak Sriprayoonsakul wrote: > >> Ok, I have a chance to run booster over 2.0.4 >> > > > Have you tried configuring booster with the help doc available at: > http://www.gluster.org/docs/index.php/BoosterConfiguration > > -Shehjar > > >> Please find the attach file for my configuration >> >> I did configure boost and try simple ls over my Gluster file system. >> Here's >> the output of ls -al >> >> [root at compute-0-3 ~]# >> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >> ls -l /gluster/www/ >> ls: /gluster/www/: Invalid argument >> ls: /gluster/www/members.pantip.com: Invalid argument >> ls: /gluster/www/cafe.pantip.com: Invalid argument >> ls: /gluster/www/admin.pantip.com: Invalid argument >> ls: /gluster/www/www.pantip.com: Invalid argument >> ls: /gluster/www/passwd3.sql: Invalid argument >> ls: /gluster/www/passwd.sql: Invalid argument >> ls: closing directory /gluster/www/: File descriptor in bad state >> total 129972 >> drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com >> drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com >> drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com >> -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql >> -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql >> drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com >> [root at compute-0-3 ~]# >> >> [root at compute-0-3 ~]# >> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >> cp /etc/issue /gluster/ >> [root at compute-0-3 ~]# >> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >> ls -l /gluster/issue >> ls: /gluster/issue: Invalid argument >> -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue >> [root at compute-0-3 ~]# >> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >> cat /gluster/issue >> CentOS release 5.3 (Final) >> Kernel \r on an \m >> >> [root at compute-0-3 ~]# >> >> >> Despite all those errors, output seems to be fine >> >> And this is what inside my booster.log >> >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] >> libglusterfsclient: path lookup failed for (/hosts) >> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] >> libglusterfsclient: path lookup failed for (/issue) >> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] >> libglusterfsclient: invalid argument: value >> >> Then, I try to LD_PRELOAD apache (prefork). I change the target from >> /gluster to /usr/home instead (the web application needs it). Then I tried >> to strace the httpd process and found that httpd crash at the points where >> httpd tried to read configuration file stored on Gluster volume (bad file >> descriptor). I will try to move this configuration file some other places >> and test again. >> >> 2009/7/31 Raghavendra G <raghavendra at gluster.com> >> >> Hi, >>> >>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak >>> Sriprayoonsakul<somsaks at gmail.com> wrote: >>> >>>> Thank you very much for you reply >>>> >>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I >>>> >>> will >>> >>>> try 2.0.4 very soon to see if it's work. >>>> >>>> For Booster, it seems not working correctly for me. Booster complains a >>>> >>> lots >>> >>>> of error with plain 'ls' command (but giving the correct output). Also, >>>> >>> with >>> >>> Can you mail those errors? >>> >>> booster, Apache process refuse to start. I will try 2.0.4 to see if it >>>> improves. If not, I will attach error log next time. >>>> >>> logs are very much appreciated. >>> >>> >>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com> >>>> >>>>> Hi Somsak, >>>>> >>>>> Sorry for the delayed reply. Below you've mentioned that you've >>>>> problems >>>>> with apache and booster. Going forward, Apache over booster will be the >>>>> preferred approach. Can you tell us what version of glusterfs you are >>>>> >>>> using? >>> >>>> And as I can understand you are using apache 2.2, am I correct? >>>>> >>>>> regards, >>>>> ----- Original Message ----- >>>>> From: "Liam Slusser" <lslusser at gmail.com> >>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com> >>>>> Cc: gluster-users at gluster.org >>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat >>>>> Subject: Re: Gluster 2.0.3 + Apache on CentOS5 >>>>> >>>> performance >>> >>>> issue >>>>> >>>>> I haven't tried an apples to apples comparison with Apache+mod_gluster >>>>> >>>> vs >>> >>>> Apache+fuse+gluster however i do run both setups. I load tested both >>>>> setups >>>>> so to verified it could handle 4x our normal daily load and left it at >>>>> that. >>>>> I didn't actually compare the two (although that might be cool to do >>>>> someday). >>>>> I really like the idea of Apache+mod_gluster as I don't have to deal >>>>> >>>> with >>> >>>> the whole fuse and mounting the filesystem. It always scares me having >>>>> >>>> a >>> >>>> public facing webserver with your whole backend fileshare mounted >>>>> >>>> locally. >>> >>>> Its very slick for serving content such as media files. We serve audio >>>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing >>>>> 200-300mbit on average daily and everything works very well. >>>>> >>>>> We run an apache+fuse+gluster setup because we need to run some >>>>> mod_perl >>>>> before serving the actual content. However performance is still very >>>>> good. >>>>> We do around 50-100 requests (all jpeg images) per second off of a >>>>> fuse >>>>> mount and everything works great. We also have a java >>>>> >>>> tomcat+fuse+gluster >>> >>>> service which does image manipulation on the fly off of a gluster mount. >>>>> >>>>> We have two backend gluster servers using replication which serve all >>>>> >>>> this >>> >>>> content. >>>>> >>>>> If you would like more information on our setup id be happy to share >>>>> offline. Just email me privately. >>>>> >>>>> thanks, >>>>> liam >>>>> >>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul >>>>> <somsaks at gmail.com>wrote: >>>>> >>>>> Oh thank you, thought noone will reply me :) >>>>>> >>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance? >>>>>> >>>>>> Also, anyone in this mailing-list have tried Apache with booster? I >>>>>> tried >>>>>> it but Apache refuse to start (just hang and freeze). >>>>>> >>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com> >>>>>> >>>>>> >>>>>> We use mod_gluster and Apache >>>>>>> 2.2 with good results. We also ran into the same issue as you that >>>>>>> >>>>>> we >>> >>>> ran out of memory past 150 threads even on a 8gig machine. We got >>>>>>> >>>>>> around >>> >>>> this by compiling Apache using mpm-worker >>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the same >>>>>>> number >>>>>>> of connections (150-200) and everything has been running smoothly. I >>>>>>> cannot >>>>>>> see any performance difference except it using way less memory. >>>>>>> liam >>>>>>> >>>>>>> >>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < >>>>>>> somsaks at gmail.com> wrote: >>>>>>> >>>>>>> Hello, >>>>>>>> >>>>>>>> We have been evaluating the choice for the new platform for a >>>>>>>> >>>>>>> webboard >>> >>>> system. >>>>>>>> The webboard is PHP scripts that generate/modify HTML page when user >>>>>>>> posting/add comment to the page, resulting topic is actually stored >>>>>>>> >>>>>>> as >>> >>>> a >>>>>>>> HTML file with all related file (file attach to the topic, etc.. >>>>>>>> )stored in >>>>>>>> its own directory for each topic. In general, the web site mostly >>>>>>>> serve a >>>>>>>> lot of small static files using Apache while using PHP to do other >>>>>>>> dynamic >>>>>>>> contents. This system has been working very well in the past, with >>>>>>>> >>>>>>> the >>> >>>> increasing page view rate, it is very likely that we will need some >>>>>>>> kind of >>>>>>>> Cluster file system as backend very soon. >>>>>>>> >>>>>>>> We have set up a test system using Grinder as stress test tool. The >>>>>>>> test >>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock >>>>>>>> Apache >>>>>>>> (prefork, since the goal is to use this with PHP), linked together >>>>>>>> with >>>>>>>> Gigabit Ethernet. We try to compare the performance of either using >>>>>>>> single >>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute of >>>>>>>> >>>>>>> 2 >>> >>>> replicated nodes) through Fuse. However, the transaction per second >>>>>>>> (TPS) >>>>>>>> result is not good. >>>>>>>> >>>>>>>> NFS (single server, sync mode) >>>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean >>>>>>>> response time = 61.63 ms >>>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 >>>>>>>> >>>>>>> ms >>> >>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms >>>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = >>>>>>>> 287.33ms >>>>>>>> >>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node) >>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = >>>>>>>> >>>>>>> 207.33ms >>> >>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = >>>>>>>> 672.67ms >>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33 >>>>>>>> (no 400-600 threads since we run out of client machine, sorry). >>>>>>>> >>>>>>>> gfsd is configured to use 32 thread of iothread as brick. gfs-client >>>>>>>> is >>>>>>>> configured to use >>>>>>>> io-cache->write-behind->readahead->distribute->replicate. >>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from >>>>>>>> Gluster >>>>>>>> web-site (build through DKMS). >>>>>>>> >>>>>>>> As the result yield, it seems that Gluster performance worse with >>>>>>>> increasing no. of client. One observation is that the glusterfs >>>>>>>> process on >>>>>>>> client is taking about 100% of CPU during all the tests. glusterfsd >>>>>>>> >>>>>>> is >>> >>>> utilizing only 70-80% of CPUs during the test time. Note that system >>>>>>>> is Dual >>>>>>>> core. >>>>>>>> >>>>>>>> I also tried using modglusterfs and not using fuse at all to serve >>>>>>>> >>>>>>> all >>> >>>> the static files and conduct another test with Grinder. The result >>>>>>>> >>>>>>> is >>> >>>> about >>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in >>>>>>>> >>>>>>> this >>> >>>> test >>>>>>>> that each Apache prefork process used more about twice more memory >>>>>>>> >>>>>>> and >>> >>>> we >>>>>>>> need to lower number of httpd processes by about half. >>>>>>>> >>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting >>>>>>>> readahead, >>>>>>>> write behind according to GlusterOptimization page didn't help much >>>>>>>> either. >>>>>>>> >>>>>>>> My question is, there seems to be bottleneck in this setup, but how >>>>>>>> can I >>>>>>>> track this? Note that, I didn't do any other optimization other than >>>>>>>> what >>>>>>>> said above. Are there any best practice configuration for using >>>>>>>> >>>>>>> Apache >>> >>>> to >>>>>>>> serve a bunch of small static files like this around? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Somsak >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>> regards, >>> -- >>> Raghavendra G >>> >>> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> > >
