Hi.
I have been working with Apache for more than 8 years, but now I'm facing a severe problem and I couldn't find a solution on internet so far.
I'm running Apache 2.2.15 32bit on a Windows 2008 R2 64bit with Service Pack 1. I always used a 32bit Apache because my CGI codes (exe files) are 32bit.
The computer has a 3.10 GHz processor and 16GB RAM. The server load is always very low. The access log showed a normal request amount when the problem happened.
Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don't know how to fix this problem.
It has been many weeks I don't change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.
By using Windows Event Viewer, I have found an error log when Apache stopped working. It's the error event:
-----------------------------------------------
Log Name: Application
Source: Application Error
Date: 6/28/2016 8:08:33 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: PictureToPeople
Description:
Faulting application name: httpd.exe, version: 2.2.15.0, time stamp: 0x4b8fed95
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58
Exception code: 0xc0000005
Fault offset: 0x00037188
Faulting process id: 0xbbc
Faulting application start time: 0x01d1cdfef0f8148c
Faulting application path: C:\Apache2.2\bin\httpd.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 3c1e0a43-3d85-11e6-aad3-001e8cf18221
Event Xml:
<Event xmlns="
http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-06-28T23:08:33.000000000Z" />
<EventRecordID>693</EventRecordID>
<Channel>Application</Channel>
<Computer>PictureToPeople</Computer>
<Security />
</System>
<EventData>
<Data>httpd.exe</Data>
<Data>2.2.15.0</Data>
<Data>4b8fed95</Data>
<Data>ntdll.dll</Data>
<Data>6.1.7601.17514</Data>
<Data>4ce7ba58</Data>
<Data>c0000005</Data>
<Data>00037188</Data>
<Data>bbc</Data>
<Data>01d1cdfef0f8148c</Data>
<Data>C:\Apache2.2\bin\httpd.exe</Data>
<Data>C:\Windows\SysWOW64\ntdll.dll</Data>
<Data>3c1e0a43-3d85-11e6-aad3-001e8cf18221</Data>
</EventData>
</Event>
-----------------------------------------------
It's what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It's possible to see the
error message "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting".
After this message, Apache never worked again.
(Previous Apache error log messages were recorded very earlier and are not related.)
-----------------------------------------------
[Tue Jun 28 20:08:30 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 20:08:31 2016] [notice] Child 3004: Exit event signaled. Child process is ending.
[Tue Jun 28 20:08:32 2016] [info] Child 3004: Accept thread exiting.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Released the start mutex
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Terminating 99 threads that failed to exit.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: All worker threads have exited.
[Tue Jun 28 20:08:32 2016] [info] [client 1.23.96.161] (OS 10038)An operation was attempted on something that is not a socket. : core_output_filter: writing data to the network
[Tue Jun 28 20:08:34 2016] [notice] Parent: Child process exited successfully.
[Tue Jun 28 20:08:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=284)
[Tue Jun 28 20:08:54 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:08:54 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 20:08:54 2016] [notice] Parent: Created child process 2468
[Tue Jun 28 20:08:54 2016] [info] Parent: Duplicating socket 308 and sending it to child process 2468
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Child process is running
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Acquired the start mutex.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting 100 worker threads.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting thread to listen on port 80.
[Tue Jun 28 20:08:56 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
-----------------------------------------------
After the error above, restarting Windows and Apache service didn't solve the problem (same error message).
After reinstalling Apache and booting the machine, the problem was still there (same error message).
So, I increased the "MaxRequestsPerChild" parameter to 300, but all worker threads were consumed in 12 seconds.
So, I increased the "MaxRequestsPerChild" parameter to 500 and rebooted the machine, but all worker threads were consumed in 9 seconds as the log shows below.
-----------------------------------------------
[Tue Jun 28 21:55:01 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 21:55:01 2016] [notice] Child 864: Exit event signaled. Child process is ending.
[Tue Jun 28 21:55:02 2016] [info] Child 864: Accept thread exiting.
[Tue Jun 28 21:55:02 2016] [notice] Child 864: Released the start mutex
[Tue Jun 28 21:55:20 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [notice] Parent: Forcing termination of child process 368
[Tue Jun 28 21:55:31 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=2252)
[Tue Jun 28 22:10:13 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 22:10:13 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 22:10:13 2016] [notice] Parent: Created child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Child process is running
[Tue Jun 28 22:10:13 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Acquired the start mutex.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting 500 worker threads.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting thread to listen on port 80.
[Tue Jun 28 22:10:22 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
-----------------------------------------------
Below you can see my complete original httpd.conf. It had been working correctly for months.
(I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data with "xxxxx"):
-----------------------------------------------
ServerRoot "C:/Apache2.2"
Listen 80
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
User daemon
Group daemon
</IfModule>
</IfModule>
ServerAdmin xxxxxxxxxxxxxxxxxxxxxxxx
ServerName xxxxxxxxxxxxxxxxxxxxxxxx:80
DocumentRoot "xxxxxxxxxxxxxxxxxxxxxxxx"
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
Options FollowSymLinks
Options +Includes
Options -Indexes
AllowOverride None
Order allow,deny
Allow from all
LimitRequestBody 10485760
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
</Directory>
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
<FilesMatch "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
<IfModule mod_headers.c>
Header always append X-Frame-Options SAMEORIGIN
</IfModule>
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 1 week"
ExpiresByType text/css "access plus 1 week"
ExpiresByType application/atom+xml "access plus 1 hour"
ExpiresByType application/rdf+xml "access plus 1 hour"
ExpiresByType application/rss+xml "access plus 1 hour"
ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/ld+json "access plus 0 seconds"
ExpiresByType application/schema+json "access plus 0 seconds"
ExpiresByType application/vnd.geo+json "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"
ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
ExpiresByType image/x-icon "access plus 1 week"
ExpiresByType text/html "access plus 1 day"
ExpiresByType application/_javascript_ "access plus 1 week"
ExpiresByType application/x-_javascript_ "access plus 1 week"
ExpiresByType text/_javascript_ "access plus 1 week"
ExpiresByType application/manifest+json "access plus 1 year"
ExpiresByType application/x-web-app-manifest+json "access plus 0 seconds"
ExpiresByType text/cache-manifest "access plus 0 seconds"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
<Directory xxxxxxxxxxxxxxxxxxxxxxxx>
ExpiresByType image/gif "access plus 0 seconds"
ExpiresByType image/jpeg "access plus 0 seconds"
ExpiresByType image/png "access plus 0 seconds"
</Directory>
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType font/eot "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType font/woff "access plus 1 month"
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType text/x-cross-domain-policy "access plus 1 week"
</IfModule>
ErrorLog "|bin/rotatelogs logs/error.%Y-%m-%d.log 50M"
LogLevel info
<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
CustomLog "|bin/rotatelogs logs/access.%Y-%m-%d---%H.log 300M" common
</IfModule>
<IfModule alias_module>
ScriptAlias /p2p/ "xxxxxxxxxxxxxxxxxxxxxxxx"
</IfModule>
<IfModule cgid_module>
</IfModule>
<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
DefaultType text/plain
<IfModule mime_module>
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType text/html .html
AddHandler server-parsed .html
</IfModule>
<IfModule mpm_winnt_module>
ThreadsPerChild 100
MaxMemFree 100
MaxRequestsPerChild 5000
</IfModule>
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www
RewriteRule (.*)
http://www.%{HTTP_HOST}$1 [L,R]
-----------------------------------------------
As shown above, my Apache stops responding almost instantly after launch.
My site doesn't connect to databases. It just serves dynamic pages by using CGI.
Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?
Could anyone give me any advice to solve this issue or at least to get a workaround?
Thanks in advance.
