- how is your iowait
? maybe apache is waiting for disk access.
#vmstat 1
procs -----------memory---------- ---swap-- -----io----
--system-- ----cpu----
r b swpd
free buff cache si
so bi bo in
cs us sy id wa
1 3 2536 698992 195040
211940 0 0
4 187 2 3 14 14 54 17
1 0 2536 698276 195064
211984 0 0
20 0 8746 9432 27 13 43 17
0 0 2536 697444 195084
211964 0 0 20
0 8428 8802 26 11 62 1
0 0 2536 697196 195088
211960 0 0
4 0 7206 6812 19 8 73 0
1 3 2536 706232 195092
211956 0 0 4
4068 5886 5305 12 7 67 14
0 82 2536 706300 195092
211956 0 0
0 844 4534 3498 9 4 24 64
4 0 2536 705516 195092
211956 0 0
0 584 7017 6394 17 10 14 59
1 0 2536 705460 195092
211956 0 0
0 0 7994 8138 18 10 72 0
0 0 2536 712564 195100
212016 0 0
8 0 6763 7027 17 8 74 0
0 0 2536 712692 195112
212004 0 0
12 0 6201 8271 15 7 77 1
1 1 2536 712256 195116
212000 0 0 20 2372
6276 5845 14 6 78 2
1 32 2536 712436 195128
211988 0 0 12 1584
6633 7025 15 7 9 69
2 4 2536 722980 195128
211988 0 0
0 592 7114 7063 16 10 10 64
1 0 2536 722796 195140
212044 0 0
32 0 7215 7184 17 9 71 4
5 0 2536 722300 195164
212088 0 0 24
168 8066 9076 25 11 61 2
1 0 2536 722180 195168
212084 0 0 4
0 7529 7952 19 10 70 0
2 27 2536 720712 195168
211268 0 0 0
3792 6718 6542 20 11 28 41
4 7 2536 730984 195168
211268 0 0 0
1616 10398 10589 44 22 0 35
3 5 2536 731864 195172
211264 0 0
4 740 8842 10712 27 14 4 54
1 2 2536 732116 195184
211320 0 0
12 40 7049 7312 17 9 43 32
This is when apache is running normally. Iowait does not
seem to be a problem, but I did not check during the crashes.
- are your sure that you don't have a
network bottleneck ? load balancer, firewall, network interface,..
We have a Twisted server running on the same machine, and it
works OK during the crashes- so it does not seem to be a network bottleneck. Plus,
system log shows only this during crashes, which indicates apache-specific bottleneck:
Oct 1 13:21:56 web10 kernel: possible SYN flooding on
port 80. Sending cookies.
Oct 1 13:22:56 web10 kernel: possible SYN flooding on
port 80. Sending cookies.
Oct 1 13:23:57 web10 kernel: possible SYN flooding on
port 80. Sending cookies.
- do
you serve static files only with you apache ? if you use php, cgi,mod_jk,mysql,...
You should have a look to them maybe they are
hanging up.
We are using mod_python. I posted to mod_python list as
well.
- if you use thread safe modules (not php) and
if you have to serve a lot of little requests, maybe you can consider to use
mpm worker instead of prefork.
This is a good idea. We will try it. Does it give
significant performance bonus? Our average request size is only 1-2K.
Alec
From: Bj
[mailto:bjkhanz@xxxxxxxxx]
Sent: Monday, October 01, 2007 3:15 AM
To: users@xxxxxxxxxxxxxxxx
Subject: Re: requests time-out under load, no warnings in
logs
Hi,
When your server is slow:
- how is your iowait
? maybe apache is waiting for disk access.
- are your sure that you don't have a network bottleneck ?
load balancer, firewall, network interface,..
- do
you serve static files only with you apache ? if you use php, cgi,mod_jk,mysql,...
You should have a look to them maybe they are
hanging up.
- if you use thread safe modules (not php) and if you have to serve
a lot of little requests, maybe you can consider to use mpm worker instead of
prefork.
--
Bj
On 10/1/07, Alec Matusis <matusis@xxxxxxxxx> wrote:
Hi Christian,
> How does your apache look memory-wise in top/htop?
When Apache is "slow", it has about 200 processes with 11-15MB RSS
each, soat least 1GB of physical RAM should be unused.
The load average drops from normal 16.0 (this is a 4 CPU machine) to only 6.0
when apache becomes "slow".
It would be interesting to see an example of a reasonable choice for
MaxMemFree, after 2 days of searching in Google I cannot find any guidelines
for setting it.
> -----Original Message-----
> From: Christian Folini [mailto:christian.folini@xxxxxxx]
> Sent: Sunday, September 30, 2007 11:55 PM
> To: users@xxxxxxxxxxxxxxxx
> Subject: Re: requests time-out under load, no warnings in
> logs
>
> Hey Alec,
>
> This looks like a tricky problem. Have you looked at the traffic
> using tcpdump or ethereal when the machine hangs? You might
> also try out mod_forensic or mod_security and add mod_security
> internal timestamps to the access-log. Using these logs you will
> get a clearer idea where it hangs.
>
> How does your apache look memory-wise in top/htop?
> Are you sure it is not an application issue - even if it shows
> serving static files. A reverse proxy setup for testing purposes
> could seperate application and static file serving to
> give you a clearer idea in this regard.
>
> These are just a lot of ideas. I do not really know how to
> solve your problem, but this would give you a clearer idea
> of the problem.
>
> I am not familiar with best practice values for MaxMemFree.
>
> regs,
>
> Christian
>
>
>
> On Sun, Sep 30, 2007 at 03:54:59PM -0700, Alec Matusis wrote:
> > We are running a busy Apache/2.0.54 server on 2.6.9 kernel, that
> suddenly
> > becomes very slow- requests either time out, or it takes 10s to serve
> a 1K
> > thumbnail.
> > It is usually correlated with load spikes, but not perfectly (by
> looking at
> > the bandwidth graph)
> > Then it requires a stop/start, and it becomes fast again.
> > This has been happening before, and it was always accompanied with
> >
> > [error] server reached MaxClients setting, consider raising the
> MaxClients
> > setting
> >
> > in the apache log.
> >
> > We raised MaxClients from 200 to 300, and now
it happens with no
> warning
> > in any logs at all, except with the usual
> >
> > web10 kernel: possible SYN flooding on port 80. Sending cookies. in
> > /var/log/messages (this has always happened when apache hung from
> overload).
> >
> > What worries me is that there are no log messages in apache logs at
> all when
> > this happens.
> >
> > This is the relevant part of httpd.conf
> >
> > ServerLimit 300
> > # we are using prefork MPM
> > <IfModule prefork.c>
> >
StartServers 10
> > MinSpareServers 5
> > MaxSpareServers 20
> >
MaxClients 300
> > MaxRequestsPerChild 10000
> >
MaxMemFree 2560
> > </IfModule>
> >
> > The server has 4GB of RAM and 4GB of swap. When apache
"crashes" i.e.
> > becomes very slow, the swap size is still 0 and vmstat shows no
> swapping.
> > I suspect the problem may be in
> >
> > MaxMemFree 2560
> >
> > but then I would expect some kind of "out of memory" errors
in the
> logs?
> > Another question: what is the good choice of MaxMemFree value? We
> want to
> > leave 512MB-1GB or RAM on this machine for other tasks.
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server
> Project.
> > See <URL: http://httpd.apache.org/userslist.html>
for more info.
> > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> >
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html>
for more info.
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:
http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
