Hi Yasser,
Coudn't it be that your DNS server/network is not following the load?
Can you still reproduce the failures with jbosshost in your /etc/hosts?
Regards,
Yann.
> ---------------------------------------------------------------------
On Thu, Apr 23, 2015 at 8:20 PM, Yasser <yarafatin@xxxxxxxxx> wrote:
> The test ran for two hours on 4/21/2015 between 8pm and 10:50pm.
> I got the SAR report generated using ksar tool and didnt find anything wrong
> with the resource usage. Memory was almost full but not swapping either. A
> lot of TCP connections which is expected at this load. CPU and IO looks
> normal. Please find attached ksar report.
>
>
> Thanks,
> Yasser
>
> On Wed, Apr 22, 2015 at 9:48 PM, Robert Webb <rwebb@xxxxxxxxxxxx> wrote:
>>
>> That is exactly what I was thinking.
>>
>> But on to your real problem....
>>
>> What do the apache server's resources look like when this starts
>> happening? Things like number of connections, tcp port utilization, cpu,
>> I/O, etc...
>>
>> Makes me wonder if there isn't something being taxed and causing the
>> server's DNS lookups to time out.
>>
>>
>>
>> On Wed, 22 Apr 2015 21:44:18 -0400
>> Yasser <yarafatin@xxxxxxxxx> wrote:
>>>
>>> Yes. The Apache servers may sound redundant with the current setup, but
>>> in the future it will be used for other purposes like proxying to
>>> different
>>> virtual ips, php hosting etc.
>>> I cannot change the enterprise suite at this point in time.
>>> So are you running through the F5, load balancing to the two Apache
>>> proxies, then load balancing with the apache proxies to your 10 back end
>>> JBOSS servers?
>>>
>>>
>>> On Wed, 22 Apr 2015 16:21:29 -0400
>>> Yasser <yarafatin@xxxxxxxxx> wrote:
>>>
>>>> I am in the midst of performance testing a web application and I get
>>>> these
>>>> errors in Apache logs
>>>>
>>>> [Sat Apr 18 03:01:01 2015] [error] [client xx.xx.xxx.253] proxy: DNS
>>>> lookup
>>>> failure for: jboss-host-name returned by /some/url, referer:
>>>> http://f5hostname/context/URI
>>>>
>>>> It occurs only at high load levels. I have seen posts where this error
>>>> occurs due to incorrect proxy configuration, but my case doesn't fall
>>>> under
>>>> that I guess. At the very end of the test I get another error - "All
>>>> workers are in error state". I think the first DNS error causes all
>>>> workers
>>>> to shutdown.
>>>>
>>>> Can someone point where I could start looking? Please let me know if you
>>>> need more details than whats mentioned below.
>>>>
>>>> *More details:*
>>>>
>>>> Apache version: 2.2
>>>>
>>>> mod_cluster: 1.2.4.Final
>>>>
>>>> I have an F5 in front of two Apache servers. There are ten JBoss servers
>>>> in
>>>> the balancer.
>>>>
>>>> Apache configuration snippet: (I am not using proxypass directive, just
>>>> modcluster)
>>>>
>>>> <IfModule mod_proxy.c>
>>>> ProxyRequests Off
>>>> ProxyVia On</IfModule>
>>>> <IfModule mpm_worker_module>
>>>> ServerLimit, StartServers and worker related attributes configured
>>>> here</IfModule>
>>>> ProxyPreserveHost OnMemManagerFile /var/cache/httpd
>>>> <VirtualHost *:1280>
>>>> ManagerBalancerName some2 (this is the same usedin jboss using
>>>> proxylist)
>>>> ........
>>>>
>>>> Logs:
>>>>
>>>> [Tue Apr 21 22:06:25 2015] [debug] mod_proxy_http.c(1973): proxy: HTTP:
>>>> serving URL http://jbosshost:8080/uri
>>>>
>>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2011): proxy: HTTP: has
>>>> acquired connection for (jbosshost)
>>>>
>>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2067): proxy: connecting
>>>> http://jbosshost:8080/uri
>>>>
>>>> [Tue Apr 21 22:06:25 2015] [error] [client 10.xxx.xxx.xxx] proxy: DNS
>>>> lookup failure for: jbosshost returned by /URI
>>>>
>>>> [Tue Apr 21 22:06:25 2015] [debug] proxy_util.c(2029): proxy: HTTP: has
>>>> released connection for (jbosshost)
>>>>
>>>> OS:
>>>>
>>>> RHEL 6.5
>>>>
>>
>>
>
>
>
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|