Hello. I have been struggling for a few months to solve a problem with
an Apache web server. First i'll try to describe the symptoms, then
give details about the configuration and what i have tried so far. I
would greatly appreciate any suggestions on how to diagnose and correct
the problem(s). I suspect there are basically 2 issues, one with
segmentation faults that cause core dumps, the other with MaxClients
being reached even if the server is not under heavy load. I don't know
if the problems are distinct or related.
Symptom 1:
After starting Apache during periods of high load the server appears to
run fine for awhile ("awhile" is highly variable but on the order of 90
minutes) then starts returning empty pages. When the empty pages start,
Apache's error_log gets filled with lines like this:
[Fri Aug 04 14:18:29 2006] [notice] child pid 16021 exit signal
Segmentation fault (11), possible coredump in /etc/httpd/core
These errors are very intermittent (occurring every few hours to every
few weeks) during periods of light load. I have had CoreDumpDirectory
defined for awhile and have amassed quite a collection of core files.
This is the first few lines of what gdb reports for one of the dumps:
(gdb) bt
#0 0x0000002a99ff2492 in preg_replace_impl (ht=Variable "ht" is not
available.)
at /usr/src/redhat/BUILD/php-4.3.9/ext/pcre/php_pcre.c:1154
#1 0x0000002a9a0ac255 in execute (op_array=0x552afe2798)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1640
#2 0x0000002a9a0a9386 in execute (op_array=0x552aff7fb8)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#3 0x0000002a9a0a9386 in execute (op_array=0x552b0e26c8)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
#4 0x0000002a9a0a9386 in execute (op_array=0x552af80db8)
at /usr/src/redhat/BUILD/php-4.3.9/Zend/zend_execute.c:1684
Symptom 2:
At other times messages about MaxClients show up. I don't believe the
server is actually running out of processes to serve requests since
even during the academic year when it is very busy requests are handled
quickly enough that there are seldom more than a dozen or two in
progress at any moment.
[Fri Aug 04 02:30:02 2006] [error] server reached MaxClients setting,
consider raising the MaxClients setting
For reference, the section of httpd.conf that controls process numbers
looks like this:
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 128
MaxRequestsPerChild 1024
I have configured the server-status handler and set a log monitoring
daemon to grab the status page from Apache when MaxClients is mentioned
in error_log. During those times the scoreboard looks like this:
119 processes closing connections
2 processes reading requests
4 processes sending replies
3 processes waiting for connections
During normal operation i consulted the scoreboard and got the following
as typical results when everything is fine:
2 closing connections
11 reading requests
3 sending replies
9 waiting for connections
Symptom 3:
A further symptom of problems is found in PostgreSQL's logs. The web
applications the server runs rely on PostgreSQL, which logs several
times per minute messages like this:
Aug 7 10:07:29 sun12 postgres[4479]: [1-1] LOG: unexpected EOF on
client connection
I believe that error occurs whenever an Apache child process dies
without properly closing its PostgreSQL connection. PostgreSQL is
configured to accept 256 connections, twice as many children as Apache
should spawn.
Symptom 4:
I don't know if this is important, but when Apache's LogLevel is cranked
up to "debug" entries like this appear in the error_log at a rate of
several per minute:
[Fri Jun 09 13:56:17 2006] [debug] util_ldap.c(1441): INIT global
mutex /tmp/filessX9mx in child 25783
Configuration:
The server is a Sun v40z, which is a dual Opteron box with 2 GB RAM. It
is running Red Hat Linux Enterprise AS 4.3 64-bit, with Apache 2.0.52,
PHP 4.3.9, and eAccelerator 0.9.4. The server's purpose is to run Horde
and Imp, with the latest versions of those packages and a few other
Horde components (Ingo, Passwd, and Turba).
Apache is installed with Red Hat's RPMs. PHP is compiled from Red Hat's
source RPMs; i added mcrypt support which is not included by default
(but needed to improve performance of Horde).
Below is a list of things i have tried that did not work. After each
test that failed to provide results i reverted to the previous
configuration.
1) Lowering MaxRequestsPerChild to 64. This increased the error rate.
2) Using Red Hat's RPMs even though they don't have mcrypt support. This
did nothing.
3) Upgrading PHP to the latest 4.4.x (x==2 at the time i tried that),
compiled from source. This had no effect.
4) Removed eAccelerator. Problems remained, but server responds to
requests a bit slower (as would be expected).
5) Running with a uniprocessor kernel. No effect on the errors.
6) Switched Apache and PHP to 32-bit. The rest of the system is 64-bit.
This also had no effect.
Over the last few months i have exchanged many messages on the Horde and
Imp mailing lists trying to track down the problems. The consensus
seems to be that Horde is not the problem, hence i turn to this list
for help instead. If you can provide any assistance, i will be very
grateful. Thank you in advance for your time and for reading this long
message.
------------------------------------------------------------------------
Dan Ramaley Dial Center 118, Drake University
Network Programmer/Analyst 2407 Carpenter Ave
+1 515 271-4540 Des Moines IA 50311 USA
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|