Ok, I have a chance to run booster over 2.0.4 Please find the attach file for my configuration I did configure boost and try simple ls over my Gluster file system. Here's the output of ls -al [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/www/ ls: /gluster/www/: Invalid argument ls: /gluster/www/members.pantip.com: Invalid argument ls: /gluster/www/cafe.pantip.com: Invalid argument ls: /gluster/www/admin.pantip.com: Invalid argument ls: /gluster/www/www.pantip.com: Invalid argument ls: /gluster/www/passwd3.sql: Invalid argument ls: /gluster/www/passwd.sql: Invalid argument ls: closing directory /gluster/www/: File descriptor in bad state total 129972 drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com [root at compute-0-3 ~]# [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/ [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/issue ls: /gluster/issue: Invalid argument -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cat /gluster/issue CentOS release 5.3 (Final) Kernel \r on an \m [root at compute-0-3 ~]# Despite all those errors, output seems to be fine And this is what inside my booster.log [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] libglusterfsclient: path lookup failed for (/hosts) [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] libglusterfsclient: path lookup failed for (/issue) [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] libglusterfsclient: invalid argument: value Then, I try to LD_PRELOAD apache (prefork). I change the target from /gluster to /usr/home instead (the web application needs it). Then I tried to strace the httpd process and found that httpd crash at the points where httpd tried to read configuration file stored on Gluster volume (bad file descriptor). I will try to move this configuration file some other places and test again. 2009/7/31 Raghavendra G <raghavendra at gluster.com> > Hi, > > On Thu, Jul 30, 2009 at 11:39 AM, Somsak > Sriprayoonsakul<somsaks at gmail.com> wrote: > > Thank you very much for you reply > > > > At the time we used 2.0.3, and yes we used stock Apache from CentOS. I > will > > try 2.0.4 very soon to see if it's work. > > > > For Booster, it seems not working correctly for me. Booster complains a > lots > > of error with plain 'ls' command (but giving the correct output). Also, > with > > Can you mail those errors? > > > booster, Apache process refuse to start. I will try 2.0.4 to see if it > > improves. If not, I will attach error log next time. > > logs are very much appreciated. > > > > > > > 2009/7/30 Raghavendra G <raghavendra at gluster.com> > >> > >> Hi Somsak, > >> > >> Sorry for the delayed reply. Below you've mentioned that you've problems > >> with apache and booster. Going forward, Apache over booster will be the > >> preferred approach. Can you tell us what version of glusterfs you are > using? > >> And as I can understand you are using apache 2.2, am I correct? > >> > >> regards, > >> ----- Original Message ----- > >> From: "Liam Slusser" <lslusser at gmail.com> > >> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com> > >> Cc: gluster-users at gluster.org > >> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat > >> Subject: Re: Gluster 2.0.3 + Apache on CentOS5 > performance > >> issue > >> > >> I haven't tried an apples to apples comparison with Apache+mod_gluster > vs > >> Apache+fuse+gluster however i do run both setups. I load tested both > >> setups > >> so to verified it could handle 4x our normal daily load and left it at > >> that. > >> I didn't actually compare the two (although that might be cool to do > >> someday). > >> I really like the idea of Apache+mod_gluster as I don't have to deal > with > >> the whole fuse and mounting the filesystem. It always scares me having > a > >> public facing webserver with your whole backend fileshare mounted > locally. > >> Its very slick for serving content such as media files. We serve audio > >> content to our CDN with a pair of Apache/mod_gluster servers - pushing > >> 200-300mbit on average daily and everything works very well. > >> > >> We run an apache+fuse+gluster setup because we need to run some mod_perl > >> before serving the actual content. However performance is still very > >> good. > >> We do around 50-100 requests (all jpeg images) per second off of a fuse > >> mount and everything works great. We also have a java > tomcat+fuse+gluster > >> service which does image manipulation on the fly off of a gluster mount. > >> > >> We have two backend gluster servers using replication which serve all > this > >> content. > >> > >> If you would like more information on our setup id be happy to share > >> offline. Just email me privately. > >> > >> thanks, > >> liam > >> > >> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul > >> <somsaks at gmail.com>wrote: > >> > >> > Oh thank you, thought noone will reply me :) > >> > > >> > Have you tried Apache + Fuse over GlusterFS? How is the performance? > >> > > >> > Also, anyone in this mailing-list have tried Apache with booster? I > >> > tried > >> > it but Apache refuse to start (just hang and freeze). > >> > > >> > 2009/7/23 Liam Slusser <lslusser at gmail.com> > >> > > >> > > >> >> We use mod_gluster and Apache > >> >> 2.2 with good results. We also ran into the same issue as you that > we > >> >> ran out of memory past 150 threads even on a 8gig machine. We got > around > >> >> this by compiling Apache using mpm-worker > >> >> (threads) instead of prefork - it uses 1/4 as much ram with the same > >> >> number > >> >> of connections (150-200) and everything has been running smoothly. I > >> >> cannot > >> >> see any performance difference except it using way less memory. > >> >> liam > >> >> > >> >> > >> >> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < > >> >> somsaks at gmail.com> wrote: > >> >> > >> >>> Hello, > >> >>> > >> >>> We have been evaluating the choice for the new platform for a > webboard > >> >>> system. > >> >>> The webboard is PHP scripts that generate/modify HTML page when user > >> >>> posting/add comment to the page, resulting topic is actually stored > as > >> >>> a > >> >>> HTML file with all related file (file attach to the topic, etc.. > >> >>> )stored in > >> >>> its own directory for each topic. In general, the web site mostly > >> >>> serve a > >> >>> lot of small static files using Apache while using PHP to do other > >> >>> dynamic > >> >>> contents. This system has been working very well in the past, with > the > >> >>> increasing page view rate, it is very likely that we will need some > >> >>> kind of > >> >>> Cluster file system as backend very soon. > >> >>> > >> >>> We have set up a test system using Grinder as stress test tool. The > >> >>> test > >> >>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock > >> >>> Apache > >> >>> (prefork, since the goal is to use this with PHP), linked together > >> >>> with > >> >>> Gigabit Ethernet. We try to compare the performance of either using > >> >>> single > >> >>> NFS server in sync mode against using 4 Gluster nodes (distribute of > 2 > >> >>> replicated nodes) through Fuse. However, the transaction per second > >> >>> (TPS) > >> >>> result is not good. > >> >>> > >> >>> NFS (single server, sync mode) > >> >>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean > >> >>> response time = 61.63 ms > >> >>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 > ms > >> >>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms > >> >>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = > >> >>> 287.33ms > >> >>> > >> >>> 4 nodes Gluster (2 distribute of replicated 2 node) > >> >>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = > 207.33ms > >> >>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = > >> >>> 672.67ms > >> >>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33 > >> >>> (no 400-600 threads since we run out of client machine, sorry). > >> >>> > >> >>> gfsd is configured to use 32 thread of iothread as brick. gfs-client > >> >>> is > >> >>> configured to use > >> >>> io-cache->write-behind->readahead->distribute->replicate. > >> >>> io-cache cache-size is 256MB. I used patched Fuse downloaded from > >> >>> Gluster > >> >>> web-site (build through DKMS). > >> >>> > >> >>> As the result yield, it seems that Gluster performance worse with > >> >>> increasing no. of client. One observation is that the glusterfs > >> >>> process on > >> >>> client is taking about 100% of CPU during all the tests. glusterfsd > is > >> >>> utilizing only 70-80% of CPUs during the test time. Note that system > >> >>> is Dual > >> >>> core. > >> >>> > >> >>> I also tried using modglusterfs and not using fuse at all to serve > all > >> >>> the static files and conduct another test with Grinder. The result > is > >> >>> about > >> >>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in > this > >> >>> test > >> >>> that each Apache prefork process used more about twice more memory > and > >> >>> we > >> >>> need to lower number of httpd processes by about half. > >> >>> > >> >>> I tried disable EnableMMAP and it didn't help much. Adjusting > >> >>> readahead, > >> >>> write behind according to GlusterOptimization page didn't help much > >> >>> either. > >> >>> > >> >>> My question is, there seems to be bottleneck in this setup, but how > >> >>> can I > >> >>> track this? Note that, I didn't do any other optimization other than > >> >>> what > >> >>> said above. Are there any best practice configuration for using > Apache > >> >>> to > >> >>> serve a bunch of small static files like this around? > >> >>> > >> >>> Regards, > >> >>> > >> >>> Somsak > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> _______________________________________________ > >> >>> Gluster-users mailing list > >> >>> Gluster-users at gluster.org > >> >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > >> >>> > >> >>> > >> >> > >> > > >> > >> _______________________________________________ > >> Gluster-users mailing list > >> Gluster-users at gluster.org > >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > > > > > regards, > -- > Raghavendra G >
