Re: Unable to fork new process

["Michele Mase'" <michele.mase@xxxxxxxxx>]

There is the possibility of a memory leak: following an old post about the same problem https://ubuntuforums.org/showthread.php?t=1505539, I tried the experiment: using the php script to keep reloading apache until apache logs an error (a simple loop that reloads apache), after 100883 reloads and 7621 seconds, the apache on the test machine hangs with the error "[error] (12)Cannot allocate memory: fork: Unable to fork new process".

Notice that the test machine is absolutely empty, there is only the web server with no activity.

On a medium/large prod. environment, some apache reload (due to config change, like add or remove a virtualhost) together with the load of the web sites causes the same error after some days. It seems to be like the following algorithm: xx apache reloads + yy load average = zz time memory leak.

I applied the sipliest solution:

changing the logrotate.d/httpd by replacin service httpd reload with service httpd restart

But the memory leak stirr remains.

Any better suggestion? Anyone ever will correct the httpd code?

Best regards

On Tue, Jan 24, 2017 at 9:52 PM, Houser, Rick <rick.houser@xxxxxxxxxxx> wrote:

Where and what is the leak?  If it’s in the workers (the parts running as non-root), perhaps just set a request limit so they keep cycling out automatically within Apache and getting replaced seamlessly with new children?  If it’s in the higher level processes (the root owned ones), that’s not going to work.  I could be mistaken, but I thought a process or thread limit also gave a message similar to that, so double check your ulimit settings.  Adding memory not helping you further implies you are running up against a software limit of some kind.

 

 

Rick Houser

Web Administration

 

From: Darryl Philip Baker [mailto:darryl.baker@northwestern.edu]
Sent: Tuesday, January 24, 2017 14:08
To: users@xxxxxxxxxxxxxxxx
Subject: RE: Unable to fork new process

 

First I am assuming you are using the Red Hat distributed version. Is upgrading to Apache httpd 2.4 from the Red Hat Software Collection an option?

 

I can think of several options to detect and automate the restart as a remediation. One would be to script a memory watcher using the free command and automatically executing the restart steps you do manually. The run the script out of cron every 5 minutes.

 

For a fix you might want to peruse the release note for Apache versions of 2.2 after 2.2.15, and the mods, for memory leak fixes and see if Red Hat has backported them into their version. If you find one that wasn’t backported time to nudge Res Hat. Maybe someone on this list more in tune with the releases can point you at one or two of these.  Or just build a newer version of Apache 2.2 from source

 

Darryl Baker

NIT - CI -DAPS

X76674

 

From: Michele Mase' [mailto:michele.mase@xxxxxxxxx]
Sent: January 24, 2017 12:10 PM
To: Httpd Users List <users@xxxxxxxxxxxxxxxx>
Subject: Unable to fork new process

 

All,
I've a apache web server with 600 web sites as reverse proxy. The average of req. is more than 100 req/sec during all the day.
The apache is a 2.2.15 release 56.el6_8.3 (RedHat 6.x system up to date) + mod_ssl.
Every 10 days, the system has problems, and in the error log i found dozen of:

 [error] (12)Cannot allocate memory: fork: Unable to fork new process

The temporary solution for me is the following: after an alert mail from the system monitoring server, I must restart the httpd using:

service httpd stop; killall httpd.worker; killall -9 httpd.worker; service httpd start

Adding more ram to the system does not help.

 

Any suggestion?

Best regards

apachectl -l
Compiled in modules:
  core.c
  worker.c
  http_core.c
  mod_so.c

apachectl -M
Loaded Modules:
 core_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 log_post_module (shared)
 evasive20_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 substitute_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 unique_id_module (shared)
 ssl_module (shared)
 reqtimeout_module (shared)
Syntax OK