HiThis seems to be quite similar to what I have seen, and others have also posted about the last couple of years.
I've done a lot of investigation, but have not found a real solution. My problem is that the threads gets stuck in "W" / "Sending reply" state.Are you using mod_php ? If so, it might be that one thread is doing work, while the other threads are waiting to acquire a lock on the php session file. When PHP session files are used, the request are processed sequentially, and not in parallel for a user.
I suggest you also use "strace" unix command to see what the different threads are doing, that might be easier than using gdb. When I use strace, I see that one thread is hanging on "poll" system call, until the timeout (Timeout Apache directive - default 300 seconds) occur. Then I see that the next thread hangs in "poll" and so on.
So if you set the "Timeout" lower, it might improve your situation.You should also run "netstat", and see what the status of the sockets are. If they are almost all in CLOSE_WAIT, I think you are seeing something similar to me.
Here are some links to similar issues, I'm not sure if it will help you : http://www.nabble.com/RE:-requests-time-out-under-load,-no-warnings-in-logs-p13065495.htmlhttp://mail-archives.apache.org/mod_mbox/httpd-users/200611.mbox/%3c20061114084333.GC10809@xxxxxxxxxxxxxxxxxxxxxxxx%3e ( and the other posts to that thread)
http://www.linuxforums.org/forum/servers/27519-alright-im-dying-here-apache-help-mysql-php.html http://osdir.com/ml/apache.mod-ssl.user/2004-10/msg00009.htmlI have also seen other posts from people with very similar problems, lots of thread stuck in a state.
What is your environment ? I.e. what version of Apache, what operating system and version, are you generating dynamic pages, and if so, what tools (for example php) are you using ?
Do you have a firewall or a load balancer in front of your web server ?Are almost all the requests from the same IP adresse ? Are almost all the request for the same URL ?
What my theory is at the moment, after throwing away some other theories, are that somehow a client creates several hundred requests for the same URL. Since in my case this happens fairly often, several times a day on a uncontroversial high traffic site, from different users, I do not think it is someone doing a DoS attack.
I think it might be a problem with a firewall or a proxy or maybe the browser (I have seen it happen for people using both Windows and Mac, both Firefox and IE), that sits between the web server and the end user. Somehow the closing of the socket/connection by the browser is not picked up by the web server. And somehow the browser issues lots of requests. I have not completely ruled out that it might be a Apache web server bug as well. But since I have no way of reproducing this, apart from observing it happening quite often, it is very hard to get any further.
So any help would be greatly appreciated. Regards Alf Hogemark Sander Temme wrote:
On Nov 7, 2007, at 9:31 AM, Andrew Rosolino wrote:I restarted apache and all was fine.. but then 20 minutes later they went back all into a reading state.. it appears as if slowly each processes goesinto the reading state?? I dont understand what the problem is.If you have gdb on the box (and have some acumen using it), you can attach to one of the processes and see if it is hanging in something obvious.If not, and your traffic is plaintext, I recommend the tcpdump approach to see what is going on traffic-wise, and what triggers the issue. I'd dump the traffic to disk (tcpdump -i ethX -s 0 -w /wherever/traffic.dump port 80) and take it over to a box with Ethereal/Wireshark for some cozy analysis.If you're being DOS attacked by trickle requests, you could try setting a very low timeout (default is 5 minutes which doesn't seem to be working for you) and perhaps use mod_evasive or somesuch to flag and firewall the bad clients.S.
--------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|