Hi Yasser, Coudn't it be that your DNS server/network is not following the load? Can you still reproduce the failures with jbosshost in your /etc/hosts? Regards, Yann. On Thu, Apr 23, 2015 at 8:20 PM, Yasser <yarafatin@xxxxxxxxx> wrote: > The test ran for two hours on 4/21/2015 between 8pm and 10:50pm. > I got the SAR report generated using ksar tool and didnt find anything wrong > with the resource usage. Memory was almost full but not swapping either. A > lot of TCP connections which is expected at this load. CPU and IO looks > normal. Please find attached ksar report. > > > Thanks, > Yasser > > On Wed, Apr 22, 2015 at 9:48 PM, Robert Webb <rwebb@xxxxxxxxxxxx> wrote: >> >> That is exactly what I was thinking. >> >> But on to your real problem.... >> >> What do the apache server's resources look like when this starts >> happening? Things like number of connections, tcp port utilization, cpu, >> I/O, etc... >> >> Makes me wonder if there isn't something being taxed and causing the >> server's DNS lookups to time out. >> >> >> >> On Wed, 22 Apr 2015 21:44:18 -0400 >> Yasser <yarafatin@xxxxxxxxx> wrote: >>> >>> Yes. The Apache servers may sound redundant with the current setup, but >>> in the future it will be used for other purposes like proxying to >>> different >>> virtual ips, php hosting etc. >>> I cannot change the enterprise suite at this point in time. >>> So are you running through the F5, load balancing to the two Apache >>> proxies, then load balancing with the apache proxies to your 10 back end >>> JBOSS servers? >>> >>> >>> On Wed, 22 Apr 2015 16:21:29 -0400 >>> Yasser <yarafatin@xxxxxxxxx> wrote: >>> >>>> I am in the midst of performance testing a web application and I get >>>> these >>>> errors in Apache logs >>>> >>>> [Sat Apr 18 03:01:01 2015] [error] [client xx.xx.xxx.253] proxy: DNS >>>> lookup >>>> failure for: jboss-host-name returned by /some/url, referer: >>>> http://f5hostname/context/URI >>>> >>>> It occurs only at high load levels. I have seen posts where this error >>>> occurs due to incorrect proxy configuration, but my case doesn't fall >>>> under >>>> that I guess. At the very end of the test I get another error - "All >>>> workers are in error state". I think the first DNS error causes all >>>> workers >>>> to shutdown. >>>> >>>> Can someone point where I could start looking? Please let me know if you >>>> need more details than whats mentioned below. >>>> >>>> *More details:* >>>> >>>> Apache version: 2.2 >>>> >>>> mod_cluster: 1.2.4.Final >>>> >>>> I have an F5 in front of two Apache servers. There are ten JBoss servers >>>> in >>>> the balancer. >>>> >>>> Apache configuration snippet: (I am not using proxypass directive, just >>>> modcluster) >>>> >>>> <IfModule mod_proxy.c> >>>> ProxyRequests Off >>>> ProxyVia On</IfModule> >>>> <IfModule mpm_worker_module> >>>> ServerLimit, StartServers and worker related attributes configured >>>> here</IfModule> >>>> ProxyPreserveHost OnMemManagerFile /var/cache/httpd >>>> <VirtualHost *:1280> >>>> ManagerBalancerName some2 (this is the same usedin jboss using >>>> proxylist) >>>> ........ >>>> >>>> Logs: >>>> >>>> [Tue Apr 21 22:06:25 2015] [debug] mod_proxy_http.c(1973): proxy: HTTP: >>>> serving URL http://jbosshost:8080/uri >>>> >>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2011): proxy: HTTP: has >>>> acquired connection for (jbosshost) >>>> >>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2067): proxy: connecting >>>> http://jbosshost:8080/uri >>>> >>>> [Tue Apr 21 22:06:25 2015] [error] [client 10.xxx.xxx.xxx] proxy: DNS >>>> lookup failure for: jbosshost returned by /URI >>>> >>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2029): proxy: HTTP: has >>>> released connection for (jbosshost) >>>> >>>> OS: >>>> >>>> RHEL 6.5 >>>> >> >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|