Somsak Sriprayoonsakul wrote: > Ok, I have a chance to run booster over 2.0.4 Have you tried configuring booster with the help doc available at: http://www.gluster.org/docs/index.php/BoosterConfiguration -Shehjar > > Please find the attach file for my configuration > > I did configure boost and try simple ls over my Gluster file system. Here's > the output of ls -al > > [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so > ls -l /gluster/www/ > ls: /gluster/www/: Invalid argument > ls: /gluster/www/members.pantip.com: Invalid argument > ls: /gluster/www/cafe.pantip.com: Invalid argument > ls: /gluster/www/admin.pantip.com: Invalid argument > ls: /gluster/www/www.pantip.com: Invalid argument > ls: /gluster/www/passwd3.sql: Invalid argument > ls: /gluster/www/passwd.sql: Invalid argument > ls: closing directory /gluster/www/: File descriptor in bad state > total 129972 > drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com > drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com > drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com > -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql > -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql > drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com > [root at compute-0-3 ~]# > > [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so > cp /etc/issue /gluster/ > [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so > ls -l /gluster/issue > ls: /gluster/issue: Invalid argument > -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue > [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so > cat /gluster/issue > CentOS release 5.3 (Final) > Kernel \r on an \m > > [root at compute-0-3 ~]# > > > Despite all those errors, output seems to be fine > > And this is what inside my booster.log > > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] > libglusterfsclient: path lookup failed for (/hosts) > [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] > libglusterfsclient: path lookup failed for (/issue) > [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] > libglusterfsclient: invalid argument: value > > Then, I try to LD_PRELOAD apache (prefork). I change the target from > /gluster to /usr/home instead (the web application needs it). Then I tried > to strace the httpd process and found that httpd crash at the points where > httpd tried to read configuration file stored on Gluster volume (bad file > descriptor). I will try to move this configuration file some other places > and test again. > > 2009/7/31 Raghavendra G <raghavendra at gluster.com> > >> Hi, >> >> On Thu, Jul 30, 2009 at 11:39 AM, Somsak >> Sriprayoonsakul<somsaks at gmail.com> wrote: >>> Thank you very much for you reply >>> >>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I >> will >>> try 2.0.4 very soon to see if it's work. >>> >>> For Booster, it seems not working correctly for me. Booster complains a >> lots >>> of error with plain 'ls' command (but giving the correct output). Also, >> with >> >> Can you mail those errors? >> >>> booster, Apache process refuse to start. I will try 2.0.4 to see if it >>> improves. If not, I will attach error log next time. >> logs are very much appreciated. >> >>> >>> 2009/7/30 Raghavendra G <raghavendra at gluster.com> >>>> Hi Somsak, >>>> >>>> Sorry for the delayed reply. Below you've mentioned that you've problems >>>> with apache and booster. Going forward, Apache over booster will be the >>>> preferred approach. Can you tell us what version of glusterfs you are >> using? >>>> And as I can understand you are using apache 2.2, am I correct? >>>> >>>> regards, >>>> ----- Original Message ----- >>>> From: "Liam Slusser" <lslusser at gmail.com> >>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com> >>>> Cc: gluster-users at gluster.org >>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat >>>> Subject: Re: Gluster 2.0.3 + Apache on CentOS5 >> performance >>>> issue >>>> >>>> I haven't tried an apples to apples comparison with Apache+mod_gluster >> vs >>>> Apache+fuse+gluster however i do run both setups. I load tested both >>>> setups >>>> so to verified it could handle 4x our normal daily load and left it at >>>> that. >>>> I didn't actually compare the two (although that might be cool to do >>>> someday). >>>> I really like the idea of Apache+mod_gluster as I don't have to deal >> with >>>> the whole fuse and mounting the filesystem. It always scares me having >> a >>>> public facing webserver with your whole backend fileshare mounted >> locally. >>>> Its very slick for serving content such as media files. We serve audio >>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing >>>> 200-300mbit on average daily and everything works very well. >>>> >>>> We run an apache+fuse+gluster setup because we need to run some mod_perl >>>> before serving the actual content. However performance is still very >>>> good. >>>> We do around 50-100 requests (all jpeg images) per second off of a fuse >>>> mount and everything works great. We also have a java >> tomcat+fuse+gluster >>>> service which does image manipulation on the fly off of a gluster mount. >>>> >>>> We have two backend gluster servers using replication which serve all >> this >>>> content. >>>> >>>> If you would like more information on our setup id be happy to share >>>> offline. Just email me privately. >>>> >>>> thanks, >>>> liam >>>> >>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul >>>> <somsaks at gmail.com>wrote: >>>> >>>>> Oh thank you, thought noone will reply me :) >>>>> >>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance? >>>>> >>>>> Also, anyone in this mailing-list have tried Apache with booster? I >>>>> tried >>>>> it but Apache refuse to start (just hang and freeze). >>>>> >>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com> >>>>> >>>>> >>>>>> We use mod_gluster and Apache >>>>>> 2.2 with good results. We also ran into the same issue as you that >> we >>>>>> ran out of memory past 150 threads even on a 8gig machine. We got >> around >>>>>> this by compiling Apache using mpm-worker >>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the same >>>>>> number >>>>>> of connections (150-200) and everything has been running smoothly. I >>>>>> cannot >>>>>> see any performance difference except it using way less memory. >>>>>> liam >>>>>> >>>>>> >>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < >>>>>> somsaks at gmail.com> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> We have been evaluating the choice for the new platform for a >> webboard >>>>>>> system. >>>>>>> The webboard is PHP scripts that generate/modify HTML page when user >>>>>>> posting/add comment to the page, resulting topic is actually stored >> as >>>>>>> a >>>>>>> HTML file with all related file (file attach to the topic, etc.. >>>>>>> )stored in >>>>>>> its own directory for each topic. In general, the web site mostly >>>>>>> serve a >>>>>>> lot of small static files using Apache while using PHP to do other >>>>>>> dynamic >>>>>>> contents. This system has been working very well in the past, with >> the >>>>>>> increasing page view rate, it is very likely that we will need some >>>>>>> kind of >>>>>>> Cluster file system as backend very soon. >>>>>>> >>>>>>> We have set up a test system using Grinder as stress test tool. The >>>>>>> test >>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock >>>>>>> Apache >>>>>>> (prefork, since the goal is to use this with PHP), linked together >>>>>>> with >>>>>>> Gigabit Ethernet. We try to compare the performance of either using >>>>>>> single >>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute of >> 2 >>>>>>> replicated nodes) through Fuse. However, the transaction per second >>>>>>> (TPS) >>>>>>> result is not good. >>>>>>> >>>>>>> NFS (single server, sync mode) >>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean >>>>>>> response time = 61.63 ms >>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 >> ms >>>>>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms >>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = >>>>>>> 287.33ms >>>>>>> >>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node) >>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = >> 207.33ms >>>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = >>>>>>> 672.67ms >>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33 >>>>>>> (no 400-600 threads since we run out of client machine, sorry). >>>>>>> >>>>>>> gfsd is configured to use 32 thread of iothread as brick. gfs-client >>>>>>> is >>>>>>> configured to use >>>>>>> io-cache->write-behind->readahead->distribute->replicate. >>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from >>>>>>> Gluster >>>>>>> web-site (build through DKMS). >>>>>>> >>>>>>> As the result yield, it seems that Gluster performance worse with >>>>>>> increasing no. of client. One observation is that the glusterfs >>>>>>> process on >>>>>>> client is taking about 100% of CPU during all the tests. glusterfsd >> is >>>>>>> utilizing only 70-80% of CPUs during the test time. Note that system >>>>>>> is Dual >>>>>>> core. >>>>>>> >>>>>>> I also tried using modglusterfs and not using fuse at all to serve >> all >>>>>>> the static files and conduct another test with Grinder. The result >> is >>>>>>> about >>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in >> this >>>>>>> test >>>>>>> that each Apache prefork process used more about twice more memory >> and >>>>>>> we >>>>>>> need to lower number of httpd processes by about half. >>>>>>> >>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting >>>>>>> readahead, >>>>>>> write behind according to GlusterOptimization page didn't help much >>>>>>> either. >>>>>>> >>>>>>> My question is, there seems to be bottleneck in this setup, but how >>>>>>> can I >>>>>>> track this? Note that, I didn't do any other optimization other than >>>>>>> what >>>>>>> said above. Are there any best practice configuration for using >> Apache >>>>>>> to >>>>>>> serve a bunch of small static files like this around? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Somsak >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >> >> regards, >> -- >> Raghavendra G >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
