Apache stopped working permanently after error "Server ran out of threads to serve requests."

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi.

I have been working with Apache for more than 8 years, but now I'm facing a severe problem and I couldn't find a solution on internet so far.

I'm running Apache 2.2.15 32bit on a Windows 2008 R2 64bit with Service Pack 1. I always used a 32bit Apache because my CGI codes (exe files) are 32bit.

The computer has a 3.10 GHz processor and 16GB RAM. The server load is always very low. The access log showed a normal request amount when the problem happened.

Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don't know how to fix this problem.

It has been many weeks I don't change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.


By using Windows Event Viewer, I have found an error log when Apache stopped working. It's the error event:

-----------------------------------------------
    Log Name:      Application
    Source:        Application Error
    Date:          6/28/2016 8:08:33 PM
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      PictureToPeople
    Description:
    Faulting application name: httpd.exe, version: 2.2.15.0, time stamp: 0x4b8fed95
    Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58
    Exception code: 0xc0000005
    Fault offset: 0x00037188
    Faulting process id: 0xbbc
    Faulting application start time: 0x01d1cdfef0f8148c
    Faulting application path: C:\Apache2.2\bin\httpd.exe
    Faulting module path: C:\Windows\SysWOW64\ntdll.dll
    Report Id: 3c1e0a43-3d85-11e6-aad3-001e8cf18221
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2016-06-28T23:08:33.000000000Z" />
        <EventRecordID>693</EventRecordID>
        <Channel>Application</Channel>
        <Computer>PictureToPeople</Computer>
        <Security />
      </System>
      <EventData>
        <Data>httpd.exe</Data>
        <Data>2.2.15.0</Data>
        <Data>4b8fed95</Data>
        <Data>ntdll.dll</Data>
        <Data>6.1.7601.17514</Data>
        <Data>4ce7ba58</Data>
        <Data>c0000005</Data>
        <Data>00037188</Data>
        <Data>bbc</Data>
        <Data>01d1cdfef0f8148c</Data>
        <Data>C:\Apache2.2\bin\httpd.exe</Data>
        <Data>C:\Windows\SysWOW64\ntdll.dll</Data>
        <Data>3c1e0a43-3d85-11e6-aad3-001e8cf18221</Data>
      </EventData>
    </Event>
-----------------------------------------------

It's what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It's possible to see the
error message "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting".
After this message, Apache never worked again.
(Previous Apache error log messages were recorded very earlier and are not related.)

-----------------------------------------------
    [Tue Jun 28 20:08:30 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
    [Tue Jun 28 20:08:31 2016] [notice] Child 3004: Exit event signaled. Child process is ending.
    [Tue Jun 28 20:08:32 2016] [info] Child 3004: Accept thread exiting.
    [Tue Jun 28 20:08:32 2016] [notice] Child 3004: Released the start mutex
    [Tue Jun 28 20:08:32 2016] [notice] Child 3004: Terminating 99 threads that failed to exit.
    [Tue Jun 28 20:08:32 2016] [notice] Child 3004: All worker threads have exited.
    [Tue Jun 28 20:08:32 2016] [info] [client 1.23.96.161] (OS 10038)An operation was attempted on something that is not a socket.  : core_output_filter: writing data to the network
    [Tue Jun 28 20:08:34 2016] [notice] Parent: Child process exited successfully.
    [Tue Jun 28 20:08:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=284)
    [Tue Jun 28 20:08:54 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
    [Tue Jun 28 20:08:54 2016] [notice] Server built: Mar  4 2010 11:27:46
    [Tue Jun 28 20:08:54 2016] [notice] Parent: Created child process 2468
    [Tue Jun 28 20:08:54 2016] [info] Parent: Duplicating socket 308 and sending it to child process 2468
    [Tue Jun 28 20:08:54 2016] [notice] Child 2468: Child process is running
    [Tue Jun 28 20:08:54 2016] [notice] Child 2468: Acquired the start mutex.
    [Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting 100 worker threads.
    [Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting thread to listen on port 80.
    [Tue Jun 28 20:08:56 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
-----------------------------------------------

After the error above, restarting Windows and Apache service didn't solve the problem (same error message).
After reinstalling Apache and booting the machine, the problem was still there (same error message).
So, I increased the "MaxRequestsPerChild" parameter to 300, but all worker threads were consumed in 12 seconds.
So, I increased the "MaxRequestsPerChild" parameter to 500 and rebooted the machine, but all worker threads were consumed in 9 seconds as the log shows below.

-----------------------------------------------
    [Tue Jun 28 21:55:01 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
    [Tue Jun 28 21:55:01 2016] [notice] Child 864: Exit event signaled. Child process is ending.
    [Tue Jun 28 21:55:02 2016] [info] Child 864: Accept thread exiting.
    [Tue Jun 28 21:55:02 2016] [notice] Child 864: Released the start mutex
    [Tue Jun 28 21:55:20 2016] [error] [client 1.23.96.161] request failed: error reading the headers
    [Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
    [Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
    [Tue Jun 28 21:55:31 2016] [notice] Parent: Forcing termination of child process 368
    [Tue Jun 28 21:55:31 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=2252)
    [Tue Jun 28 22:10:13 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
    [Tue Jun 28 22:10:13 2016] [notice] Server built: Mar  4 2010 11:27:46
    [Tue Jun 28 22:10:13 2016] [notice] Parent: Created child process 1452
    [Tue Jun 28 22:10:13 2016] [notice] Child 1452: Child process is running
    [Tue Jun 28 22:10:13 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1452
    [Tue Jun 28 22:10:13 2016] [notice] Child 1452: Acquired the start mutex.
    [Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting 500 worker threads.
    [Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting thread to listen on port 80.
    [Tue Jun 28 22:10:22 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
-----------------------------------------------

Below you can see my complete original httpd.conf. It had been working correctly for months.
(I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data with "xxxxx"):


-----------------------------------------------
    ServerRoot "C:/Apache2.2"
   
    Listen 80
   
    LoadModule actions_module modules/mod_actions.so
    LoadModule alias_module modules/mod_alias.so
    LoadModule asis_module modules/mod_asis.so
    LoadModule auth_basic_module modules/mod_auth_basic.so
    LoadModule authn_default_module modules/mod_authn_default.so
    LoadModule authn_file_module modules/mod_authn_file.so
    LoadModule authz_default_module modules/mod_authz_default.so
    LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
    LoadModule authz_host_module modules/mod_authz_host.so
    LoadModule authz_user_module modules/mod_authz_user.so
    LoadModule autoindex_module modules/mod_autoindex.so
    LoadModule cgi_module modules/mod_cgi.so
    LoadModule deflate_module modules/mod_deflate.so
    LoadModule dir_module modules/mod_dir.so
    LoadModule env_module modules/mod_env.so
    LoadModule expires_module modules/mod_expires.so
    LoadModule headers_module modules/mod_headers.so
    LoadModule include_module modules/mod_include.so
    LoadModule isapi_module modules/mod_isapi.so
    LoadModule log_config_module modules/mod_log_config.so
    LoadModule mime_module modules/mod_mime.so
    LoadModule negotiation_module modules/mod_negotiation.so
    LoadModule rewrite_module modules/mod_rewrite.so
    LoadModule setenvif_module modules/mod_setenvif.so
   
    <IfModule !mpm_netware_module>
    <IfModule !mpm_winnt_module>
    User daemon
    Group daemon
   
    </IfModule>
    </IfModule>
   
   
    ServerAdmin xxxxxxxxxxxxxxxxxxxxxxxx
   
      ServerName xxxxxxxxxxxxxxxxxxxxxxxx:80
   
   
    DocumentRoot "xxxxxxxxxxxxxxxxxxxxxxxx"
   
    <Directory />
        Options FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
    </Directory>
   
   
    <Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
        Options FollowSymLinks
        Options +Includes
        Options -Indexes
   
        AllowOverride None
   
        Order allow,deny
        Allow from all
   
        LimitRequestBody 10485760
   
        SetOutputFilter DEFLATE
        SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
   
    </Directory>
   
    <IfModule dir_module>
        DirectoryIndex index.html
    </IfModule>
   
    <FilesMatch "^\.ht">
        Order allow,deny
        Deny from all
        Satisfy All
    </FilesMatch>
   
    <IfModule mod_headers.c>
        Header always append X-Frame-Options SAMEORIGIN
    </IfModule>
   
    <IfModule mod_expires.c>
   
        ExpiresActive on
        ExpiresDefault                                      "access plus 1 week"
   
        ExpiresByType text/css                              "access plus 1 week"
   
        ExpiresByType application/atom+xml                  "access plus 1 hour"
        ExpiresByType application/rdf+xml                   "access plus 1 hour"
        ExpiresByType application/rss+xml                   "access plus 1 hour"
   
        ExpiresByType application/json                      "access plus 0 seconds"
        ExpiresByType application/ld+json                   "access plus 0 seconds"
        ExpiresByType application/schema+json               "access plus 0 seconds"
        ExpiresByType application/vnd.geo+json              "access plus 0 seconds"
        ExpiresByType application/xml                       "access plus 0 seconds"
        ExpiresByType text/xml                              "access plus 0 seconds"
   
        ExpiresByType image/vnd.microsoft.icon              "access plus 1 week"
        ExpiresByType image/x-icon                          "access plus 1 week"
   
        ExpiresByType text/html                             "access plus 1 day"
   
        ExpiresByType application/_javascript_                "access plus 1 week"
        ExpiresByType application/x-_javascript_              "access plus 1 week"
        ExpiresByType text/_javascript_                       "access plus 1 week"
   
        ExpiresByType application/manifest+json             "access plus 1 year"
   
        ExpiresByType application/x-web-app-manifest+json   "access plus 0 seconds"
        ExpiresByType text/cache-manifest                   "access plus 0 seconds"
   
        ExpiresByType audio/ogg                             "access plus 1 month"
        ExpiresByType image/bmp                             "access plus 1 month"
        ExpiresByType image/gif                             "access plus 1 month"
        ExpiresByType image/jpeg                            "access plus 1 month"
        ExpiresByType image/png                             "access plus 1 month"
        ExpiresByType image/svg+xml                         "access plus 1 month"
        ExpiresByType video/mp4                             "access plus 1 month"
        ExpiresByType video/ogg                             "access plus 1 month"
        ExpiresByType video/webm                            "access plus 1 month"
   
      <Directory xxxxxxxxxxxxxxxxxxxxxxxx>
          ExpiresByType image/gif                             "access plus 0 seconds"
          ExpiresByType image/jpeg                            "access plus 0 seconds"
          ExpiresByType image/png                             "access plus 0 seconds"
      </Directory>
   
   
   
        ExpiresByType application/vnd.ms-fontobject         "access plus 1 month"
        ExpiresByType font/eot                              "access plus 1 month"
   
        ExpiresByType font/opentype                         "access plus 1 month"
   
        ExpiresByType application/x-font-ttf                "access plus 1 month"
   
        ExpiresByType application/font-woff                 "access plus 1 month"
        ExpiresByType application/x-font-woff               "access plus 1 month"
        ExpiresByType font/woff                             "access plus 1 month"
   
        ExpiresByType application/font-woff2                "access plus 1 month"
   
        ExpiresByType text/x-cross-domain-policy            "access plus 1 week"
   
    </IfModule>
   
   
    ErrorLog "|bin/rotatelogs logs/error.%Y-%m-%d.log 50M"
   
    LogLevel info
   
    <IfModule log_config_module>
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
        LogFormat "%h %l %u %t \"%r\" %>s %b" common
   
        <IfModule logio_module>
          LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
        </IfModule>
   
        CustomLog "|bin/rotatelogs logs/access.%Y-%m-%d---%H.log 300M" common
   
    </IfModule>
   
    <IfModule alias_module>
   
   
        ScriptAlias /p2p/ "xxxxxxxxxxxxxxxxxxxxxxxx"
   
    </IfModule>
   
    <IfModule cgid_module>
    </IfModule>
   
    <Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
        AllowOverride None
        Options None
        Order allow,deny
        Allow from all
    </Directory>
   
    DefaultType text/plain
   
    <IfModule mime_module>
        TypesConfig conf/mime.types
   
        AddType application/x-compress .Z
        AddType application/x-gzip .gz .tgz
   
        AddType text/html .html
        AddHandler server-parsed .html
    </IfModule>
   
   
    <IfModule mpm_winnt_module>
        ThreadsPerChild      100
        MaxMemFree           100
        MaxRequestsPerChild  5000
    </IfModule>
   
   
    <IfModule ssl_module>
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
    </IfModule>
   
   
    RewriteEngine On
    RewriteCond %{HTTP_HOST} !^www
    RewriteRule (.*) http://www.%{HTTP_HOST}$1 [L,R]
-----------------------------------------------

As shown above, my Apache stops responding almost instantly after launch.

My site doesn't connect to databases. It just serves dynamic pages by using CGI.

Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?

Could anyone give me any advice to solve this issue or at least to get a workaround?

Thanks in advance.

[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux