Somsak Sriprayoonsakul wrote: > I have run html read benchmark test using exactly the same old testset. > > - Runing normal httpd (prefork) over NFS-sync option - Average TPS ~ 1500, > Peak TPS ~ 3500 > > - Running normal httpd (prefork) using booster - Average TPS < 500, Peak TPS > ~ 1200, slight improvements but each HTTPD still eat up about 50MB+ (keep > increasing) memory instead of about 20MB without booster. I have to cut the > number of httpd process by half, but still the process used up some swap > space. > > - Running httpd.worker using booster - The benchmark result is very low and > error rate is very high. There seems to be some trouble in this mode. > - Here's what I did > - Changed from httpd to httpd.worker in /etc/sysconfig/httpd > - Disable PHP4 > - LD_PRELOAD in /etc/init.d/httpd then start httpd > - The HTTPD start correctly and seems to work ok, but not all the time. Two > consecutive wget's on exactly the same URL yield different results Hi I can see a few reasons why this could happen with recent releases. If possible, could you please try the same test with the release below. In this release, I've fixed a few bugs that could result in behaviour seen in your tests. http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-2.0.6rc2.tar.gz Thanks Shehjar > > [root at compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null > http://c0-3/cafe/siam/topic/F7800428/F7800428.html > --15:52:12-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html > Resolving c0-3... 10.1.255.251 > Connecting to c0-3|10.1.255.251|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 264373 (258K) [text/html] > Saving to: `/dev/null' > > 100%[====================================================================================================================>] > 264,373 --.-K/s in 0.003s > > 15:52:12 (83.3 MB/s) - `/dev/null' saved [264373/264373] > > [root at compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null > http://c0-3/cafe/siam/topic/F7800428/F7800428.html > --15:52:15-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html > Resolving c0-3... 10.1.255.251 > Connecting to c0-3|10.1.255.251|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 0 [text/html] > Saving to: `/dev/null' > > [ > <=> > ] 0 --.-K/s in 0s > > 15:52:15 (0.00 B/s) - `/dev/null' saved [0/0] > > [root at compute-0-9 ~]# > > (Note the returned content-length. Returned contents, when corrected, is > ok) > > Here's the error log in booster log file > > (First wget) > > [2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry] > libglusterfsclient: VMP Entry found: /usr/home/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/ > [2009-08-03 15:36:21] D > [libglusterfsclient-dentry.c:381:libgf_client_path_lookup] > libglusterfsclient: resolved path(/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to > 2626532/2650924 > [2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry] > libglusterfsclient: VMP Entry found: /usr/home/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/ > [2009-08-03 15:36:21] D > [libglusterfsclient-dentry.c:381:libgf_client_path_lookup] > libglusterfsclient: resolved path(/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to > 2626532/2650924 > > (second wget) > > [2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry] > libglusterfsclient: VMP Entry found: /usr/home/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/ > [2009-08-03 15:36:26] D > [libglusterfsclient-dentry.c:381:libgf_client_path_lookup] > libglusterfsclient: resolved path(/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to > 2626532/2650924 > [2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry] > libglusterfsclient: VMP Entry found: /usr/home/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/ > [2009-08-03 15:36:26] D > [libglusterfsclient-dentry.c:381:libgf_client_path_lookup] > libglusterfsclient: resolved path(/www/ > www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to > 2626532/2650924 > > In server logs, there are just connect-destroy-connection messages in the > log. Nothing particularily wrong to me. > > No such failure occur with Apache+Prefork. > > 2009/8/2 Somsak Sriprayoonsakul <somsaks at gmail.com> > >> After moving the embed configuration file out from glusterfs, now httpd >> boot up ok and the web seems to work now. >> >> However, comment posting is not working. It seems that the code that do the >> html modification is not working in GlusterFS context. I found that the code >> modify web page locally instead. So I think workaround for my case is to >> mount glusterfs with fuse at the same path as booster. >> >> I will give it another benchmark and see how it goes. >> >> 2009/8/2 Somsak Sriprayoonsakul <somsaks at gmail.com> >> >> Ok, I have a chance to run booster over 2.0.4 >>> Please find the attach file for my configuration >>> >>> I did configure boost and try simple ls over my Gluster file system. >>> Here's the output of ls -al >>> >>> [root at compute-0-3 ~]# >>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/www/ >>> ls: /gluster/www/: Invalid argument >>> ls: /gluster/www/members.pantip.com: Invalid argument >>> ls: /gluster/www/cafe.pantip.com: Invalid argument >>> ls: /gluster/www/admin.pantip.com: Invalid argument >>> ls: /gluster/www/www.pantip.com: Invalid argument >>> ls: /gluster/www/passwd3.sql: Invalid argument >>> ls: /gluster/www/passwd.sql: Invalid argument >>> ls: closing directory /gluster/www/: File descriptor in bad state >>> total 129972 >>> drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com >>> drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com >>> drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com >>> -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql >>> -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql >>> drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com >>> [root at compute-0-3 ~]# >>> >>> [root at compute-0-3 ~]# >>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/ >>> [root at compute-0-3 ~]# >>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/issue >>> ls: /gluster/issue: Invalid argument >>> -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue >>> [root at compute-0-3 ~]# >>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cat /gluster/issue >>> CentOS release 5.3 (Final) >>> Kernel \r on an \m >>> >>> [root at compute-0-3 ~]# >>> >>> >>> Despite all those errors, output seems to be fine >>> >>> And this is what inside my booster.log >>> >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat] >>> libglusterfsclient: path lookup failed for (/hosts) >>> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat] >>> libglusterfsclient: path lookup failed for (/issue) >>> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr] >>> libglusterfsclient: invalid argument: value >>> >>> Then, I try to LD_PRELOAD apache (prefork). I change the target from >>> /gluster to /usr/home instead (the web application needs it). Then I tried >>> to strace the httpd process and found that httpd crash at the points where >>> httpd tried to read configuration file stored on Gluster volume (bad file >>> descriptor). I will try to move this configuration file some other places >>> and test again. >>> >>> 2009/7/31 Raghavendra G <raghavendra at gluster.com> >>> >>> Hi, >>>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak >>>> Sriprayoonsakul<somsaks at gmail.com> wrote: >>>>> Thank you very much for you reply >>>>> >>>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I >>>> will >>>>> try 2.0.4 very soon to see if it's work. >>>>> >>>>> For Booster, it seems not working correctly for me. Booster complains a >>>> lots >>>>> of error with plain 'ls' command (but giving the correct output). Also, >>>> with >>>> >>>> Can you mail those errors? >>>> >>>>> booster, Apache process refuse to start. I will try 2.0.4 to see if it >>>>> improves. If not, I will attach error log next time. >>>> logs are very much appreciated. >>>> >>>>> >>>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com> >>>>>> Hi Somsak, >>>>>> >>>>>> Sorry for the delayed reply. Below you've mentioned that you've >>>> problems >>>>>> with apache and booster. Going forward, Apache over booster will be >>>> the >>>>>> preferred approach. Can you tell us what version of glusterfs you are >>>> using? >>>>>> And as I can understand you are using apache 2.2, am I correct? >>>>>> >>>>>> regards, >>>>>> ----- Original Message ----- >>>>>> From: "Liam Slusser" <lslusser at gmail.com> >>>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com> >>>>>> Cc: gluster-users at gluster.org >>>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat >>>>>> Subject: Re: Gluster 2.0.3 + Apache on CentOS5 >>>> performance >>>>>> issue >>>>>> >>>>>> I haven't tried an apples to apples comparison with Apache+mod_gluster >>>> vs >>>>>> Apache+fuse+gluster however i do run both setups. I load tested both >>>>>> setups >>>>>> so to verified it could handle 4x our normal daily load and left it at >>>>>> that. >>>>>> I didn't actually compare the two (although that might be cool to do >>>>>> someday). >>>>>> I really like the idea of Apache+mod_gluster as I don't have to deal >>>> with >>>>>> the whole fuse and mounting the filesystem. It always scares me >>>> having a >>>>>> public facing webserver with your whole backend fileshare mounted >>>> locally. >>>>>> Its very slick for serving content such as media files. We serve >>>> audio >>>>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing >>>>>> 200-300mbit on average daily and everything works very well. >>>>>> >>>>>> We run an apache+fuse+gluster setup because we need to run some >>>> mod_perl >>>>>> before serving the actual content. However performance is still very >>>>>> good. >>>>>> We do around 50-100 requests (all jpeg images) per second off of a >>>> fuse >>>>>> mount and everything works great. We also have a java >>>> tomcat+fuse+gluster >>>>>> service which does image manipulation on the fly off of a gluster >>>> mount. >>>>>> We have two backend gluster servers using replication which serve all >>>> this >>>>>> content. >>>>>> >>>>>> If you would like more information on our setup id be happy to share >>>>>> offline. Just email me privately. >>>>>> >>>>>> thanks, >>>>>> liam >>>>>> >>>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul >>>>>> <somsaks at gmail.com>wrote: >>>>>> >>>>>>> Oh thank you, thought noone will reply me :) >>>>>>> >>>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance? >>>>>>> >>>>>>> Also, anyone in this mailing-list have tried Apache with booster? I >>>>>>> tried >>>>>>> it but Apache refuse to start (just hang and freeze). >>>>>>> >>>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com> >>>>>>> >>>>>>> >>>>>>>> We use mod_gluster and Apache >>>>>>>> 2.2 with good results. We also ran into the same issue as you that >>>> we >>>>>>>> ran out of memory past 150 threads even on a 8gig machine. We got >>>> around >>>>>>>> this by compiling Apache using mpm-worker >>>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the >>>> same >>>>>>>> number >>>>>>>> of connections (150-200) and everything has been running smoothly. >>>> I >>>>>>>> cannot >>>>>>>> see any performance difference except it using way less memory. >>>>>>>> liam >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul < >>>>>>>> somsaks at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> We have been evaluating the choice for the new platform for a >>>> webboard >>>>>>>>> system. >>>>>>>>> The webboard is PHP scripts that generate/modify HTML page when >>>> user >>>>>>>>> posting/add comment to the page, resulting topic is actually >>>> stored as >>>>>>>>> a >>>>>>>>> HTML file with all related file (file attach to the topic, etc.. >>>>>>>>> )stored in >>>>>>>>> its own directory for each topic. In general, the web site mostly >>>>>>>>> serve a >>>>>>>>> lot of small static files using Apache while using PHP to do other >>>>>>>>> dynamic >>>>>>>>> contents. This system has been working very well in the past, with >>>> the >>>>>>>>> increasing page view rate, it is very likely that we will need >>>> some >>>>>>>>> kind of >>>>>>>>> Cluster file system as backend very soon. >>>>>>>>> >>>>>>>>> We have set up a test system using Grinder as stress test tool. >>>> The >>>>>>>>> test >>>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock >>>>>>>>> Apache >>>>>>>>> (prefork, since the goal is to use this with PHP), linked together >>>>>>>>> with >>>>>>>>> Gigabit Ethernet. We try to compare the performance of either >>>> using >>>>>>>>> single >>>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute >>>> of 2 >>>>>>>>> replicated nodes) through Fuse. However, the transaction per >>>> second >>>>>>>>> (TPS) >>>>>>>>> result is not good. >>>>>>>>> >>>>>>>>> NFS (single server, sync mode) >>>>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, >>>> mean >>>>>>>>> response time = 61.63 ms >>>>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 >>>> ms >>>>>>>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms >>>>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = >>>>>>>>> 287.33ms >>>>>>>>> >>>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node) >>>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = >>>> 207.33ms >>>>>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = >>>>>>>>> 672.67ms >>>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = >>>> 931.33 >>>>>>>>> (no 400-600 threads since we run out of client machine, sorry). >>>>>>>>> >>>>>>>>> gfsd is configured to use 32 thread of iothread as brick. >>>> gfs-client >>>>>>>>> is >>>>>>>>> configured to use >>>>>>>>> io-cache->write-behind->readahead->distribute->replicate. >>>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from >>>>>>>>> Gluster >>>>>>>>> web-site (build through DKMS). >>>>>>>>> >>>>>>>>> As the result yield, it seems that Gluster performance worse with >>>>>>>>> increasing no. of client. One observation is that the glusterfs >>>>>>>>> process on >>>>>>>>> client is taking about 100% of CPU during all the tests. >>>> glusterfsd is >>>>>>>>> utilizing only 70-80% of CPUs during the test time. Note that >>>> system >>>>>>>>> is Dual >>>>>>>>> core. >>>>>>>>> >>>>>>>>> I also tried using modglusterfs and not using fuse at all to serve >>>> all >>>>>>>>> the static files and conduct another test with Grinder. The result >>>> is >>>>>>>>> about >>>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in >>>> this >>>>>>>>> test >>>>>>>>> that each Apache prefork process used more about twice more memory >>>> and >>>>>>>>> we >>>>>>>>> need to lower number of httpd processes by about half. >>>>>>>>> >>>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting >>>>>>>>> readahead, >>>>>>>>> write behind according to GlusterOptimization page didn't help >>>> much >>>>>>>>> either. >>>>>>>>> >>>>>>>>> My question is, there seems to be bottleneck in this setup, but >>>> how >>>>>>>>> can I >>>>>>>>> track this? Note that, I didn't do any other optimization other >>>> than >>>>>>>>> what >>>>>>>>> said above. Are there any best practice configuration for using >>>> Apache >>>>>>>>> to >>>>>>>>> serve a bunch of small static files like this around? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Somsak >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>>>> >>>> >>>> regards, >>>> -- >>>> Raghavendra G >>>> >>> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
