Re: [users@httpd] Apache Sometimes Ignoring MaxSpareServers, Running Out of Memory

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 9/22/06, Dave Mabe <davemabe@xxxxxxxxx> wrote:
On 9/22/06, Joshua Slive <joshua@xxxxxxxx> wrote:
> On 9/22/06, Dave Mabe <davemabe@xxxxxxxxx> wrote:
> > I'm running Apache from the apache-perl Debian package out of stable.
> >
> > Every so often (just infrequently enough to be very annoying), Apache will
> > all of the sudden ignore the MaxSpareServers and start up more and more
> > Apache child processes until it brings the server down.  During normal
> > operation, it works just fine.  But when it gets in this state, it brings
> > the server to its knees and I have to reboot.
> >
> > I'm using this version of apache-perl: 1.3.33-6sarge1
> >
> > Any ideas about how to troubleshoot and fix this problem?
>
> What does server-status look like during these times (see mod_status)?

OK, the problem happened again and I was able to pull some information
from server-status.

First of all, here's the appropriate part of my config:

MinSpareServers 3
MaxSpareServers 5
StartServers 3
MaxClients 50
MaxRequestsPerChild 30

I've attached a few files:
* server-status.normal.htm - the output from server-status during
normal operation
* server-status.problem.htm - output from server-status as soon as I
detected the problem
* server-status.problem.escalating.htm - output from server-status a
few minutes later
* process.list.txt - output from ps -ef|grep apache during the problem

As you can see from process.list.txt, Apache is completely ignoring
MaxSpareServers and creating a ton of child processes until it
completely runs out of memory.  Sometimes it gets in this state a
couple times a day, other times it will go for 10 days without the
problem occuring.

Any idea what is going on and how I might be able to fix it or
troubleshoot further?

Thanks!
Dave
Title: Apache Status

Apache Server Status for www.stocktickr.com

Server Version: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
Server Built: Sep 6 2005 13:12:34

Current Time: Friday, 22-Sep-2006 14:56:43 EDT
Restart Time: Friday, 22-Sep-2006 14:40:41 EDT
Parent Server Generation: 2
Server uptime: 16 minutes 2 seconds
Total accesses: 138 - Total Traffic: 232 kB
CPU Usage: u39.4 s1.52 cu.01 cs.02 - 4.26% CPU load
.143 requests/sec - 246 B/second - 1721 B/request
2 requests currently being processed, 3 idle servers
_W_W_...........................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

SrvPIDAccMCPU SSReqConnChildSlotClientVHostRequest
0-245560/28/28_ 11.66718060.00.030.03 66.249.72.69stocktickr.comGET /alibawla/closed/ HTTP/1.1
1-245570/26/26W 7.664300.00.030.03 24.163.71.161stocktickr.comGET /server-status HTTP/1.1
2-245580/26/26_ 6.542740.00.040.04 24.163.71.161stocktickr.comGET /server-status HTTP/1.1
3-245600/23/23W 7.797500.00.070.07 68.78.145.86stocktickr.comGET /stock/SLB/ HTTP/1.1
4-245620/26/26_ 5.41431670.00.030.03 204.212.175.30stocktickr.comGET /rss HTTP/1.0
5-2-0/0/4. 0.017100.00.000.02 65.2.117.174(unavailable)GET /i/y.png HTTP/1.1
6-2-0/0/1. 0.006800.00.000.000 65.2.117.174(unavailable)GET /i/comment.png HTTP/1.1
7-2-0/0/1. 0.0039530.00.000.00 65.2.117.174stocktickr.comGET /i/go.png HTTP/1.1
8-2-0/0/1. 0.003941170.00.000.000 65.2.117.174stocktickr.comGET /i/transmit.png HTTP/1.1
9-2-0/0/1. 0.01394260.00.000.000 65.2.117.174stocktickr.comGET /i/lightbulb.png HTTP/1.1
10-2-0/0/1. 1.8738724690.00.000.00 72.30.132.122stocktickr.comGET /stock/BRKR/ HTTP/1.0

SrvChild Server number - generation
PIDOS process ID
AccNumber of accesses this connection / this child / this slot
MMode of operation
CPUCPU usage, number of seconds
SSSeconds since beginning of most recent request
ReqMilliseconds required to process most recent request
ConnKilobytes transferred this connection
ChildMegabytes transferred this child
SlotTotal megabytes transferred this slot

Apache/1.3.33 Server at www.stocktickr.com Port 80
Title: Apache Status

Apache Server Status for www.stocktickr.com

Server Version: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
Server Built: Sep 6 2005 13:12:34

Current Time: Saturday, 23-Sep-2006 21:31:11 EDT
Restart Time: Saturday, 23-Sep-2006 06:12:33 EDT
Parent Server Generation: 0
Server uptime: 15 hours 18 minutes 38 seconds
Total accesses: 6344 - Total Traffic: 5.2 MB
CPU Usage: u33.35 s1.94 cu.01 cs.01 - .0641% CPU load
.115 requests/sec - 98 B/second - 856 B/request
30 requests currently being processed, 3 idle servers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW___...............................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

SrvPIDAccMCPU SSReqConnChildSlotClientVHostRequest
0-064550/18/1256W 8.6139900.00.001.07 66.249.72.69stocktickr.comGET /slaffont/short/ HTTP/1.1
1-064310/33/1236W 5.1955200.00.010.89 68.122.72.162stocktickr.comGET /rss/me/tag/PEG%20Below%201/ HTTP/1.1
2-064780/7/1218W 2.7622000.00.001.14 72.30.107.158stocktickr.comGET /stock/LMIA/ HTTP/1.0
3-065000/0/1219W 7.8856400.00.001.00 140.247.115.206stocktickr.comGET /stock/rhi HTTP/1.1
4-064560/17/1139W 6.3650400.00.000.84 65.214.44.80stocktickr.comGET /cma6/short/ HTTP/1.0
5-064830/4/231W 0.0061700.00.000.16 140.247.115.206stocktickr.comGET /stock/ryl HTTP/1.1
6-065030/0/17W 0.0156300.00.000.03 140.247.115.206stocktickr.comGET /stock/spf HTTP/1.1
7-065040/0/12W 0.0156300.00.000.03 140.247.115.206stocktickr.comGET /stock/kg HTTP/1.1
8-065070/1/8W 1.3756200.00.000.03 207.7.148.45stocktickr.comGET /rss/alibawla HTTP/1.1
9-065080/0/4W 0.5956200.00.000.000 140.247.115.206stocktickr.comGET /stock/bmd HTTP/1.1
10-065090/0/2W 0.9856200.00.000.000 140.247.115.206stocktickr.comGET /stock/nx HTTP/1.1
11-065100/0/1W 0.6656200.00.000.00 140.247.115.206stocktickr.comGET /stock/elnk HTTP/1.1
12-065110/0/1W 0.8956100.00.000.00 140.247.115.206stocktickr.comGET /maoxian HTTP/1.1
13-065120/0/0W 0.005615802593480.00.000.00 140.247.115.206stocktickr.comGET /stock/adbe HTTP/1.1
14-065130/0/0W 0.005615802592920.00.000.00 140.247.115.206stocktickr.comGET /idempotent HTTP/1.1
15-065140/0/0W 0.005615802592680.00.000.00 140.247.115.206stocktickr.comGET /stock/ges HTTP/1.1
16-065150/0/0W 0.005615802271820.00.000.00 72.30.102.21stocktickr.comGET /r/the/docs/signup/ HTTP/1.0
17-065410/0/0W 0.005235800605210.00.000.00 66.249.72.69stocktickr.comGET /stock/NXG/ HTTP/1.1
18-065420/0/0W 0.005095801399060.00.000.00 72.30.111.219stocktickr.comGET /r/id/stock/Q HTTP/1.0
19-065430/0/0W 0.005045801046220.00.000.00 72.30.97.217stocktickr.comGET /stock/GMTC/ HTTP/1.0
20-065450/0/0W 0.004455800605220.00.000.00 204.212.175.30stocktickr.comGET /rss HTTP/1.0
21-065490/0/0W 0.001715798660060.00.000.00 72.30.134.29stocktickr.comGET /tag/long/?alerts=1 HTTP/1.0
22-065670/0/0W 0.002225797291560.00.000.00 216.27.176.211stocktickr.comGET /rss/tag/MaoxianChat/ HTTP/1.1
23-065700/0/0W 0.001825797291490.00.000.00 72.30.132.215stocktickr.comGET /stock/UTI/ HTTP/1.0
24-065710/0/0W 0.001745797291490.00.000.00 207.7.148.45stocktickr.comGET /rss/alibawla HTTP/1.1
25-065780/0/0W 0.00315797287950.00.000.00 24.163.71.161stocktickr.comGET /admin/index.html HTTP/1.1
26-065790/0/0W 0.00305797276000.00.000.00 204.212.175.30stocktickr.comGET /rss HTTP/1.0
27-065800/0/0W 0.00305797275720.00.000.00 65.214.44.29stocktickr.comGET /rss/davelandry/ HTTP/1.1
28-065810/0/0W 0.00285797265670.00.000.00 72.30.101.222stocktickr.comGET /add/ASGRE/ HTTP/1.0
29-065820/0/0W 0.00-335796991460.00.000.00 24.163.71.161stocktickr.comGET /server-status HTTP/1.1
30-065830/0/0W 0.00285796776810.00.000.00 72.14.199.15stocktickr.comGET /rss/stock/AAPL HTTP/1.1
31-065840/0/0W 0.00285796751160.00.000.00 65.54.188.146stocktickr.comGET /add/FRO/ HTTP/1.0

SrvChild Server number - generation
PIDOS process ID
AccNumber of accesses this connection / this child / this slot
MMode of operation
CPUCPU usage, number of seconds
SSSeconds since beginning of most recent request
ReqMilliseconds required to process most recent request
ConnKilobytes transferred this connection
ChildMegabytes transferred this child
SlotTotal megabytes transferred this slot

Apache/1.3.33 Server at www.stocktickr.com Port 80
Title: Apache Status

Apache Server Status for www.stocktickr.com

Server Version: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
Server Built: Sep 6 2005 13:12:34

Current Time: Saturday, 23-Sep-2006 21:35:13 EDT
Restart Time: Saturday, 23-Sep-2006 06:12:33 EDT
Parent Server Generation: 0
Server uptime: 15 hours 22 minutes 40 seconds
Total accesses: 6346 - Total Traffic: 5.2 MB
CPU Usage: u28.11 s10.26 cu.01 cs.01 - .0693% CPU load
.115 requests/sec - 98 B/second - 858 B/request
40 requests currently being processed, 0 idle servers
WSWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWSS........................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"L" Logging, "G" Gracefully finishing, "." Open slot with no current process

SrvPIDAccMCPU SSReqConnChildSlotClientVHostRequest
0-064550/18/1256W 8.6164100.00.001.07 66.249.72.69stocktickr.comGET /slaffont/short/ HTTP/1.1
1-066060/0/1236S 5.191400.00.000.89 68.122.72.162(unavailable)GET /rss/me/tag/PEG%20Below%201/ HTTP/1.1
2-064780/7/1218W 2.7646200.00.001.14 72.30.107.158stocktickr.comGET /stock/LMIA/ HTTP/1.0
3-065001/1/1220W 10.84137932770.00.001.00 140.247.115.206stocktickr.comGET /stock/rhi HTTP/1.1
4-064560/17/1139W 6.3674600.00.000.84 65.214.44.80stocktickr.comGET /cma6/short/ HTTP/1.0
5-064830/4/231W 0.0085900.00.000.16 140.247.115.206stocktickr.comGET /stock/ryl HTTP/1.1
6-065030/0/17W 0.0180500.00.000.03 140.247.115.206stocktickr.comGET /stock/spf HTTP/1.1
7-065040/0/12W 0.0180500.00.000.03 140.247.115.206stocktickr.comGET /stock/kg HTTP/1.1
8-065070/1/8W 1.3780400.00.000.03 207.7.148.45stocktickr.comGET /rss/alibawla HTTP/1.1
9-065080/0/4W 0.59-800.00.000.000 140.247.115.206stocktickr.comGET /stock/bmd HTTP/1.1
10-065090/0/2W 0.9880400.00.000.000 140.247.115.206stocktickr.comGET /stock/nx HTTP/1.1
11-065100/0/1W 0.6680400.00.000.00 140.247.115.206stocktickr.comGET /stock/elnk HTTP/1.1
12-065110/0/1W 0.8980300.00.000.00 140.247.115.206stocktickr.comGET /maoxian HTTP/1.1
13-065120/0/0W 0.008035802593480.00.000.00 140.247.115.206stocktickr.comGET /stock/adbe HTTP/1.1
14-065130/0/0W 0.008035802592920.00.000.00 140.247.115.206stocktickr.comGET /idempotent HTTP/1.1
15-065140/0/0W 0.008035802592680.00.000.00 140.247.115.206stocktickr.comGET /stock/ges HTTP/1.1
16-065150/0/0W 0.008035802271820.00.000.00 72.30.102.21stocktickr.comGET /r/the/docs/signup/ HTTP/1.0
17-065410/0/0W 0.007655800605210.00.000.00 66.249.72.69stocktickr.comGET /stock/NXG/ HTTP/1.1
18-065420/0/0W 0.007515801399060.00.000.00 72.30.111.219stocktickr.comGET /r/id/stock/Q HTTP/1.0
19-065430/0/0W 0.007465801046220.00.000.00 72.30.97.217stocktickr.comGET /stock/GMTC/ HTTP/1.0
20-065450/0/0W 0.006875800605220.00.000.00 204.212.175.30stocktickr.comGET /rss HTTP/1.0
21-065490/0/0W 0.004135798660060.00.000.00 72.30.134.29stocktickr.comGET /tag/long/?alerts=1 HTTP/1.0
22-065670/0/0W 0.004645797291560.00.000.00 216.27.176.211stocktickr.comGET /rss/tag/MaoxianChat/ HTTP/1.1
23-065700/0/0W 0.004245797291490.00.000.00 72.30.132.215stocktickr.comGET /stock/UTI/ HTTP/1.0
24-065710/0/0W 0.004165797291490.00.000.00 207.7.148.45stocktickr.comGET /rss/alibawla HTTP/1.1
25-065780/0/0W 0.002735797287950.00.000.00 24.163.71.161stocktickr.comGET /admin/index.html HTTP/1.1
26-065790/0/0W 0.002725797276000.00.000.00 204.212.175.30stocktickr.comGET /rss HTTP/1.0
27-065800/0/0W 0.002725797275720.00.000.00 65.214.44.29stocktickr.comGET /rss/davelandry/ HTTP/1.1
28-065810/0/0W 0.002705797265670.00.000.00 72.30.101.222stocktickr.comGET /add/ASGRE/ HTTP/1.0
29-065820/1/1W 0.125400.00.020.02 72.30.97.212stocktickr.comGET / HTTP/1.0
30-065830/0/0W 0.002705796776810.00.000.00 72.14.199.15stocktickr.comGET /rss/stock/AAPL HTTP/1.1
31-065840/0/0W 0.002705796751160.00.000.00 65.54.188.146stocktickr.comGET /add/FRO/ HTTP/1.0
32-065850/0/0W 0.001435795983570.00.000.00 72.30.129.151stocktickr.comGET /stock/SFNT/ HTTP/1.0
33-065860/0/0W 0.001115795661320.00.000.00 207.7.148.45stocktickr.comGET /rss/alibawla HTTP/1.1
34-065870/0/0W 0.002185795663700.00.000.00 72.30.129.72stocktickr.comGET /grandcanuck/long/ HTTP/1.0
35-065950/0/0W 0.001215794700450.00.000.00 209.191.87.219stocktickr.comGET / HTTP/1.0
36-065960/0/0W 0.001115794700450.00.000.00 204.212.175.30stocktickr.comGET /rss HTTP/1.0
37-065980/0/0W 0.00-85794700460.00.000.00 24.163.71.161stocktickr.comGET /server-status HTTP/1.1
38-066090/0/0S 0.001300.00.000.00 (unavailable)
39-066100/0/0S 0.001300.00.000.00 (unavailable)


SrvChild Server number - generation
PIDOS process ID
AccNumber of accesses this connection / this child / this slot
MMode of operation
CPUCPU usage, number of seconds
SSSeconds since beginning of most recent request
ReqMilliseconds required to process most recent request
ConnKilobytes transferred this connection
ChildMegabytes transferred this child
SlotTotal megabytes transferred this slot

Apache/1.3.33 Server at www.stocktickr.com Port 80
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx

[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux