Joshua Slive wrote:
On 6/10/05, Rhesa Rozendaal <perl&nntp@xxxxxxxxx> wrote:
>
- You are sure that this isn't caused by some deliberate or accidental DoS? Have you examined netstat during these periods to see what is hitting apache? Does raising MaxClients help (providing you have enough memory)? If you are able to connect at all to apache, the output of the server-status page might be helpful.
- It looks very much like an accidental DoS. It's not deliberate I think, or I would expect it to be more sustained. My script restarts apache after a minute, and then it all works fine again. My suspicion is something like a bad-behaving browser opening up tons of connections at once. Oddly enough, we stress-tested the entire setup extensively (with apachebench) and it passed with flying colors, even with 1000 concurrent connections and for as much as 100000 requests.
- Yes, netstat. I wish I had added that to my reporting earlier, so I could tell you. I have been looking at it the first few times, but didn't immediately see anything suspicious. I'll report back on that when it happens again.
- I doubt raising MaxClients will help much. We are going to add another 1 or 2 GB and triple the amount of clients just to be safe though.
- I can connect fine, but never get any response. Every request I make times out. The server-status report immediately prior to when Apache goes dumb does show a sharp increase in BusyServers.
- A wild guess would be to try changing the AcceptMutex. If the clients get stock here, it could cause your server to look dead.
That doesn't ring a bell at all. I'll have a look in the docs to see if I can turn up anything useful. Sounds like you're on to something here.
- For debugging, you can start by strace'ing the children, but you may also need to try attaching to them with a debugger to see what they are really up to.
I'm afraid that's a bit out of my range of possibilities. This is a live server and I can't go tinkering too much with it I'm afraid. And I haven't been able to replicate it on another machine yet.
- mod_log_forensic is good for identifying if there is something particular about a request that is causing problems: http://httpd.apache.org/docs-2.0/mod/mod_log_forensic.html
Good suggestion. More logging is always good. In any case, thanks for your feedback! Rhesa --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|