I may have send the wrong file. Here is what exactly put in booster.conf in 3 servers /etc/glusterfs/glusterfs-client.vol /usr/home glusterfs subvolume=cache,logfile=/var/log/glusterfs/usrhome.log BTW, the problem occur randomly on the same node. Sometime I get the file, sometimes it's not. 2009/8/14 Raghavendra G <raghavendra at gluster.com> > Hi Somsak, > > Sorry for the delayed reply. > > I've tried to reproduce the issue you mentioned with 2.0.6 release but > without success. Both siege and wget report no inconsistencies in file size. > My configuration is httpd-2.2.11 (with mpm-worker), glusterfs 2.0.6. > > I also noticed that the virtual mount point (vmp) in your booster config is > /gluster. I am curious to know your what is DOCUMENT_ROOT configured to in > your httpd.conf. Because with DOCUMENT_ROOT configured to /usr/htdocs, the > vmp should be /usr/htdocs/gluster. For your configuration to work, the > DOCUMENT_ROOT should be '/'. > > Also, I am not finding the vmp specified in the url specified to wget. With > DOCUMENT_ROOT '/usr/htdocs' and vmp '/usr/htdocs/gluster', a wget of the > form, > > wget http:host/gluster/file > > will fetch the file from glusterfs mount point. On 'ls' on the gluster > mount point (mounted using same volume specification file as in > booster-config) you should be able to see 'file' listed. > > regards, > > > On Thu, Aug 6, 2009 at 9:24 AM, Somsak Sriprayoonsakul <somsaks at gmail.com>wrote: > >> The behavior is still the same >> >> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" >> http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> --12:20:33-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> Resolving c0-3... 10.1.255.251 >> Connecting to c0-3|10.1.255.251|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 264373 (258K) [text/html] >> Saving to: `/dev/null' >> >> 100%[====================================================================================================================>] >> 264,373 --.-K/s in 0.003s >> >> 12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373] >> >> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" >> http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> --12:20:35-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> Resolving c0-3... 10.1.255.251 >> Connecting to c0-3|10.1.255.251|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 0 [text/html] >> Saving to: `/dev/null' >> >> [ >> <=> >> ] 0 --.-K/s in 0s >> >> 12:20:35 (0.00 B/s) - `/dev/null' saved [0/0] >> >> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" >> http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> --12:20:38-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> Resolving c0-3... 10.1.255.251 >> Connecting to c0-3|10.1.255.251|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 0 [text/html] >> Saving to: `/dev/null' >> >> [ >> <=> >> ] 0 --.-K/s in 0s >> >> 12:20:38 (0.00 B/s) - `/dev/null' saved [0/0] >> >> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local" >> http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> --12:20:39-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html >> Resolving c0-3... 10.1.255.251 >> Connecting to c0-3|10.1.255.251|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 264373 (258K) [text/html] >> Saving to: `/dev/null' >> >> 100%[====================================================================================================================>] >> 264,373 --.-K/s in 0.003s >> >> 12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373] >> >> [root at compute-0-0 ~]# >> >> >> 4 consequtive wget yield different content length. Note that, this only >> happened with worker + booster, prefork has no problem. >> >> And the web server seems to use correct version of glusterfs >> >> [root at compute-0-3 ~]# glusterfs -V >> glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 >> Repository revision: v2.0.5-13-g5e3ca25 >> Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com> >> GlusterFS comes with ABSOLUTELY NO WARRANTY. >> You may redistribute copies of GlusterFS under the terms of the GNU >> General Public License. >> [root at compute-0-3 ~]# >> >> And all glusterfsd too >> >> [root at fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6 >> compute-0-6: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 >> compute-0-8: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 >> compute-0-9: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 >> compute-0-7: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03 >> [root at fe1 ~]# >> >> Don't sure if there is anyway to query version from running glusterfsd, >> but I did /etc/init.d/glusterfsd restart already. >> >> 2009/8/3 Shehjar Tikoo <shehjart at gluster.com> >> >>> Somsak Sriprayoonsakul wrote: >>> >>>> Ok, I have a chance to run booster over 2.0.4 >>>> >>> >>> >>> Have you tried configuring booster with the help doc available at: >>> http://www.gluster.org/docs/index.php/BoosterConfiguration >>> >>> -Shehjar >>> >>> >>>> Please find the attach file for my configuration >>>> >>>> I did configure boost and try simple ls over my Gluster file system. >>>> Here's >>>> the output of ls -al >>>> >>>> [root at compute-0-3 ~]# >>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >>>> ls -l /gluster/www/ >>>> ls: /gluster/www/: Invalid argument >>>> ls: /gluster/www/members.pantip.com: Invalid argument >>>> ls: /gluster/www/cafe.pantip.com: Invalid argument >>>> ls: /gluster/www/admin.pantip.com: Invalid argument >>>> ls: /gluster/www/www.pantip.com: Invalid argument >>>> ls: /gluster/www/passwd3.sql: Invalid argument >>>> ls: /gluster/www/passwd.sql: Invalid argument >>>> ls: closing directory /gluster/www/: File descriptor in bad state >>>> total 129972 >>>> drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com >>>> drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com >>>> drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com >>>> -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql >>>> -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql >>>> drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com >>>> [root at compute-0-3 ~]# >>>> >>>> [root at compute-0-3 ~]# >>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >>>> cp /etc/issue /gluster/ >>>> [root at compute-0-3 ~]# >>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >>>> ls -l /gluster/issue >>>> ls: /gluster/issue: Invalid argument >>>> -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue >>>> [root at compute-0-3 ~]# >>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so >>>> cat /gluster/issue >>>> CentOS release 5.3 (Final) >>>> Kernel \r on an \m >>>> >>>> [root at compute-0-3 ~]# >>>> >>>> >>>> Despite all those errors, output seems to be fine >>>> >>>> And this is what inside my booster.log >>>> >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] >>>> libglusterfsclient: path lookup failed for (/hosts) >>>> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] >>>> libglusterfsclient: path lookup failed for (/issue) >>>> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>>> libglusterfsclient: invalid argument: value >>>> >>>> Then, I try to LD_PRELOAD apache (prefork). I change the target from >>>> /gluster to /usr/home instead (the web application needs it). Then I >>>> tried >>>> to strace the httpd process and found that httpd crash at the points >>>> where >>>> httpd tried to read configuration file stored on Gluster volume (bad >>>> file >>>> descriptor). I will try to move this configuration file some other >>>> places >>>> and test again. >>>> >>>> 2009/7/31 Raghavendra G <raghavendra at gluster.com> >>>> >>>> Hi, >>>>> >>>>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak >>>>> Sriprayoonsakul<somsaks at gmail.com> wrote: >>>>> >>>>>> Thank you very much for you reply >>>>>> >>>>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I >>>>>> >>>>> will >>>>> >>>>>> try 2.0.4 very soon to see if it's work. >>>>>> >>>>>> For Booster, it seems not working correctly for me. Booster complains >>>>>> a >>>>>> >>>>> lots >>>>> >>>>>> of error with plain 'ls' command (but giving the correct output). >>>>>> Also, >>>>>> >>>>> with >>>>> >>>>> Can you mail those errors? >>>>> >>>>> booster, Apache process refuse to start. I will try 2.0.4 to see if it >>>>>> improves. If not, I will attach error log next time. >>>>>> >>>>> logs are very much appreciated. >>>>> >>>>> >>>>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com> >>>>>> >>>>>>> Hi Somsak, >>>>>>> >>>>>>> Sorry for the delayed reply. Below you've mentioned that you've >>>>>>> problems >>>>>>> with apache and booster. Going forward, Apache over booster will be >>>>>>> the >>>>>>> preferred approach. Can you tell us what version of glusterfs you are >>>>>>> >>>>>> using? >>>>> >>>>>> And as I can understand you are using apache 2.2, am I correct? >>>>>>> >>>>>>> regards, >>>>>>> ----- Original Message ----- >>>>>>> From: "Liam Slusser" <lslusser at gmail.com> >>>>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com> >>>>>>> Cc: gluster-users at gluster.org >>>>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / >>>>>>> Muscat >>>>>>> Subject: Re: Gluster 2.0.3 + Apache on CentOS5 >>>>>>> >>>>>> performance >>>>> >>>>>> issue >>>>>>> >>>>>>> I haven't tried an apples to apples comparison with >>>>>>> Apache+mod_gluster >>>>>>> >>>>>> vs >>>>> >>>>>> Apache+fuse+gluster however i do run both setups. I load tested both >>>>>>> setups >>>>>>> so to verified it could handle 4x our normal daily load and left it >>>>>>> at >>>>>>> that. >>>>>>> I didn't actually compare the two (although that might be cool to do >>>>>>> someday). >>>>>>> I really like the idea of Apache+mod_gluster as I don't have to deal >>>>>>> >>>>>> with >>>>> >>>>>> the whole fuse and mounting the filesystem. It always scares me >>>>>>> having >>>>>>> >>>>>> a >>>>> >>>>>> public facing webserver with your whole backend fileshare mounted >>>>>>> >>>>>> locally. >>>>> >>>>>> Its very slick for serving content such as media files. We serve >>>>>>> audio >>>>>>> content to our CDN with a pair of Apache/mod_gluster servers - >>>>>>> pushing >>>>>>> 200-300mbit on average daily and everything works very well. >>>>>>> >>>>>>> We run an apache+fuse+gluster setup because we need to run some >>>>>>> mod_perl >>>>>>> before serving the actual content. However performance is still very >>>>>>> good. >>>>>>> We do around 50-100 requests (all jpeg images) per second off of a >>>>>>> fuse >>>>>>> mount and everything works great. We also have a java >>>>>>> >>>>>> tomcat+fuse+gluster >>>>> >>>>>> service which does image manipulation on the fly off of a gluster >>>>>>> mount. >>>>>>> >>>>>>> We have two backend gluster servers using replication which serve all >>>>>>> >>>>>> this >>>>> >>>>>> content. >>>>>>> >>>>>>> If you would like more information on our setup id be happy to share >>>>>>> offline. Just email me privately. >>>>>>> >>>>>>> thanks, >>>>>>> liam >>>>>>> >>>>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul >>>>>>> <somsaks at gmail.com>wrote: >>>>>>> >>>>>>> Oh thank you, thought noone will reply me :) >>>>>>>> >>>>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance? >>>>>>>> >>>>>>>> Also, anyone in this mailing-list have tried Apache with booster? I >>>>>>>> tried >>>>>>>> it but Apache refuse to start (just hang and freeze). >>>>>>>> >>>>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com> >>>>>>>> >>>>>>>> >>>>>>>> We use mod_gluster and Apache >>>>>>>>> 2.2 with good results. We also ran into the same issue as you that >>>>>>>>> >>>>>>>> we >>>>> >>>>>> ran out of memory past 150 threads even on a 8gig machine. We got >>>>>>>>> >>>>>>>> around >>>>> >>>>>> this by compiling Apache using mpm-worker >>>>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the >>>>>>>>> same >>>>>>>>> number >>>>>>>>> of connections (150-200) and everything has been running smoothly. >>>>>>>>> I >>>>>>>>> cannot >>>>>>>>> see any performance difference except it using way less memory. >>>>>>>>> liam >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < >>>>>>>>> somsaks at gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> We have been evaluating the choice for the new platform for a >>>>>>>>>> >>>>>>>>> webboard >>>>> >>>>>> system. >>>>>>>>>> The webboard is PHP scripts that generate/modify HTML page when >>>>>>>>>> user >>>>>>>>>> posting/add comment to the page, resulting topic is actually >>>>>>>>>> stored >>>>>>>>>> >>>>>>>>> as >>>>> >>>>>> a >>>>>>>>>> HTML file with all related file (file attach to the topic, etc.. >>>>>>>>>> )stored in >>>>>>>>>> its own directory for each topic. In general, the web site mostly >>>>>>>>>> serve a >>>>>>>>>> lot of small static files using Apache while using PHP to do other >>>>>>>>>> dynamic >>>>>>>>>> contents. This system has been working very well in the past, with >>>>>>>>>> >>>>>>>>> the >>>>> >>>>>> increasing page view rate, it is very likely that we will need some >>>>>>>>>> kind of >>>>>>>>>> Cluster file system as backend very soon. >>>>>>>>>> >>>>>>>>>> We have set up a test system using Grinder as stress test tool. >>>>>>>>>> The >>>>>>>>>> test >>>>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock >>>>>>>>>> Apache >>>>>>>>>> (prefork, since the goal is to use this with PHP), linked together >>>>>>>>>> with >>>>>>>>>> Gigabit Ethernet. We try to compare the performance of either >>>>>>>>>> using >>>>>>>>>> single >>>>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute >>>>>>>>>> of >>>>>>>>>> >>>>>>>>> 2 >>>>> >>>>>> replicated nodes) through Fuse. However, the transaction per second >>>>>>>>>> (TPS) >>>>>>>>>> result is not good. >>>>>>>>>> >>>>>>>>>> NFS (single server, sync mode) >>>>>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, >>>>>>>>>> mean >>>>>>>>>> response time = 61.63 ms >>>>>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 >>>>>>>>>> >>>>>>>>> ms >>>>> >>>>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms >>>>>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = >>>>>>>>>> 287.33ms >>>>>>>>>> >>>>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node) >>>>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = >>>>>>>>>> >>>>>>>>> 207.33ms >>>>> >>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = >>>>>>>>>> 672.67ms >>>>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = >>>>>>>>>> 931.33 >>>>>>>>>> (no 400-600 threads since we run out of client machine, sorry). >>>>>>>>>> >>>>>>>>>> gfsd is configured to use 32 thread of iothread as brick. >>>>>>>>>> gfs-client >>>>>>>>>> is >>>>>>>>>> configured to use >>>>>>>>>> io-cache->write-behind->readahead->distribute->replicate. >>>>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from >>>>>>>>>> Gluster >>>>>>>>>> web-site (build through DKMS). >>>>>>>>>> >>>>>>>>>> As the result yield, it seems that Gluster performance worse with >>>>>>>>>> increasing no. of client. One observation is that the glusterfs >>>>>>>>>> process on >>>>>>>>>> client is taking about 100% of CPU during all the tests. >>>>>>>>>> glusterfsd >>>>>>>>>> >>>>>>>>> is >>>>> >>>>>> utilizing only 70-80% of CPUs during the test time. Note that system >>>>>>>>>> is Dual >>>>>>>>>> core. >>>>>>>>>> >>>>>>>>>> I also tried using modglusterfs and not using fuse at all to serve >>>>>>>>>> >>>>>>>>> all >>>>> >>>>>> the static files and conduct another test with Grinder. The result >>>>>>>>>> >>>>>>>>> is >>>>> >>>>>> about >>>>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in >>>>>>>>>> >>>>>>>>> this >>>>> >>>>>> test >>>>>>>>>> that each Apache prefork process used more about twice more memory >>>>>>>>>> >>>>>>>>> and >>>>> >>>>>> we >>>>>>>>>> need to lower number of httpd processes by about half. >>>>>>>>>> >>>>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting >>>>>>>>>> readahead, >>>>>>>>>> write behind according to GlusterOptimization page didn't help >>>>>>>>>> much >>>>>>>>>> either. >>>>>>>>>> >>>>>>>>>> My question is, there seems to be bottleneck in this setup, but >>>>>>>>>> how >>>>>>>>>> can I >>>>>>>>>> track this? Note that, I didn't do any other optimization other >>>>>>>>>> than >>>>>>>>>> what >>>>>>>>>> said above. Are there any best practice configuration for using >>>>>>>>>> >>>>>>>>> Apache >>>>> >>>>>> to >>>>>>>>>> serve a bunch of small static files like this around? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Somsak >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>> regards, >>>>> -- >>>>> Raghavendra G >>>>> >>>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>> >>> >>> >> > > > -- > Raghavendra G > >
