Re: requests time-out under load, no warnings in logs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Hi,

When your server is slow:
   - how is your iowait ? maybe apache is waiting for disk access.
   - are your sure that you don't have a network bottleneck ? load balancer, firewall, network interface,..
  - do you serve static files only with you apache ? if you use php, cgi,mod_jk,mysql,... You should have a look to them maybe they are hanging up.
  - if you use thread safe modules (not php) and if you have to serve a lot of little requests, maybe you can consider to use mpm worker instead of prefork.

--
Bj


On 10/1/07, Alec Matusis <matusis@xxxxxxxxx> wrote:
Hi Christian,

> How does your apache look memory-wise in top/htop?

When Apache is "slow", it has about 200 processes with 11-15MB RSS each, soat least 1GB of physical RAM should be unused.
The load average drops from normal 16.0 (this is a 4 CPU machine) to only 6.0 when apache becomes "slow".


It would be interesting to see an example of a reasonable choice for MaxMemFree, after 2 days of searching in Google I cannot find any guidelines for setting it.



> -----Original Message-----
> From: Christian Folini [mailto:christian.folini@xxxxxxx]
> Sent: Sunday, September 30, 2007 11:55 PM
> To: users@xxxxxxxxxxxxxxxx
> Subject: Re: requests time-out under load, no warnings in
> logs
>
> Hey Alec,
>
> This looks like a tricky problem. Have you looked at the traffic
> using tcpdump or ethereal when the machine hangs? You might
> also try out mod_forensic or mod_security and add mod_security
> internal timestamps to the access-log. Using these logs you will
> get a clearer idea where it hangs.
>
> How does your apache look memory-wise in top/htop?
> Are you sure it is not an application issue - even if it shows
> serving static files. A reverse proxy setup for testing purposes
> could seperate application and static file serving to
> give you a clearer idea in this regard.
>
> These are just a lot of ideas. I do not really know how to
> solve your problem, but this would give you a clearer idea
> of the problem.
>
> I am not familiar with best practice values for MaxMemFree.
>
> regs,
>
> Christian
>
>
>
> On Sun, Sep 30, 2007 at 03:54:59PM -0700, Alec Matusis wrote:
> > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that
> suddenly
> > becomes very slow- requests either time out, or it takes 10s to serve
> a 1K
> > thumbnail.
> > It is usually correlated with load spikes, but not perfectly (by
> looking at
> > the bandwidth graph)
> > Then it requires a stop/start, and it becomes fast again.
> > This has been happening before, and it was always accompanied with
> >
> > [error] server reached MaxClients setting, consider raising the
> MaxClients
> > setting
> >
> > in the apache log.
> >
> > We raised MaxClients    from 200 to 300, and now it happens with no
> warning
> > in any logs at all, except with the usual
> >
> > web10 kernel: possible SYN flooding on port 80. Sending cookies. in
> > /var/log/messages (this has always happened when apache hung from
> overload).
> >
> > What worries me is that there are no log messages in apache logs at
> all when
> > this happens.
> >
> > This is the relevant part of httpd.conf
> >
> > ServerLimit 300
> > # we are using prefork MPM
> > <IfModule prefork.c>
> > StartServers          10
> > MinSpareServers       5
> > MaxSpareServers       20
> > MaxClients            300
> > MaxRequestsPerChild   10000
> > MaxMemFree            2560
> > </IfModule>
> >
> > The server has 4GB of RAM and 4GB of swap. When apache "crashes" i.e.
> > becomes very slow, the swap size is still 0 and vmstat shows no
> swapping.
> > I suspect the problem may be in
> >
> > MaxMemFree   2560
> >
> > but then I would expect some kind of "out of memory" errors in the
> logs?
> > Another question: what is the good choice of MaxMemFree value? We
> want to
> > leave 512MB-1GB or RAM on this machine for other tasks.
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL: http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> >    "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>    "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL: http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx



[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux