Thanks Michele. I was going to investigate Apache HTTPD 2.4 for this issue since we use “systemctl reload httpd” every time we modify our configuration. I’m not
sure if we will ever get to 10^6 reload before the system is rebooted for a new kernel or a new version of systemd. We are using RHEL7 and the wonderfully well written systemd. :-P
Darryl Baker
Sr. System Administrator
Northwestern |
Information Technology
www.it.northwestern.edu
From: Michele Mase' [mailto:michele.mase@xxxxxxxxx]
Sent: January 27, 2017 2:03 AM
To: Httpd Users List <users@xxxxxxxxxxxxxxxx>
Subject: Re: [users@httpd] Unable to fork new process
(12)Cannot allocate memory: AH00283: fork: Unable to fork new process again!
I suppose that apache2.4 too shoul be affected by the same problem. Apache 2.4 crashed, in the same manner of apache2.2. The difference is between the number of reloads: there are more than 10^6 times of reloads to crash apache 2.4.x. I started the loop on
Yesterday, at 13:49 and the system crashed at 20:03, after 6h14minutes.
The impact of apache 2.4.x could be moderate, it's difficult to plan 10^6 of httpd reloads.
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid 140259330607072] AH00295: caught SIGTERM, shutting down
[Thu Jan 26 13:56:47.388838 2017] [core:notice] [pid 18517:tid 140237258659808] SELinux policy enabled; httpd running as context unconfined_u:system_r:initrc_t:s0
[Thu Jan 26 13:56:47.442743 2017] [suexec:notice] [pid 18517:tid 140237258659808] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Thu Jan 26 13:56:47.673152 2017] [ssl:warn] [pid 18517:tid 140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:48.745210 2017] [auth_digest:notice] [pid 18825:tid 140237258659808] AH01757: generating secret for digest authentication ...
[Thu Jan 26 13:56:48.776278 2017] [lbmethod_heartbeat:notice] [pid 18825:tid 140237258659808] AH02282: No slotmem from mod_heartmonitor
[Thu Jan 26 13:56:48.835965 2017] [ssl:warn] [pid 18825:tid 140237258659808] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Jan 26 13:56:49.040532 2017] [ssl:warn] [pid 18825:tid 140237258659808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Jan 26 13:56:49.107329 2017] [mpm_worker:notice] [pid 18825:tid 140237258659808] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips configured -- resuming normal operations
[Thu Jan 26 13:56:49.112274 2017] [core:notice] [pid 18825:tid 140237258659808] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd'
[root@kvm164 httpd]# tail -f /opt/rh/httpd24/root/var/log/httpd/error_log
[Thu Jan 26 13:49:33.450510 2017] [mpm_worker:notice] [pid 18097:tid 140259330607072] AH00295: caught SIGTERM, shutting down
......
and finally the crash
......
[Thu Jan 26 20:03:11.052290 2017] [mpm_worker:error] [pid 19594:tid 140421173676000] (12)Cannot allocate memory: AH00283: fork: Unable to fork new process
[Thu Jan 26 20:03:11.126645 2017] [mpm_worker:notice] [pid 19594:tid 140421173676000] AH00292: Apache/2.4.6 (Red Hat) OpenSSL/1.0.0-fips configured -- resuming normal operations
[Thu Jan 26 20:03:11.126958 2017] [core:notice] [pid 19594:tid 140421173676000] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd'
[Thu Jan 26 20:03:11.187777 2017] [mpm_worker:notice] [pid 19594:tid 140421173676000] AH00298: SIGHUP received. Attempting to restart
(12)Cannot allocate memory: AH00104: unable to start piped log program '/usr/sbin/rotatelogs /etc/httpd/logs/www.example.com/www.example.com_ssl_access_log_192.168.122.165
86400'
[Thu Jan 26 20:03:15.825822 2017] [:emerg] [pid 19594:tid 140421173676000] AH00019: Unable to open logs, exiting
Maybe somebody sould fix the code.
Best Regards
Michele MAsè
On Thu, Jan 26, 2017 at 5:58 PM, Michele Mase' <michele.mase@xxxxxxxxx> wrote:
Ok, I've just tried the loop within apache2.4.x and I confirm it is not affected by the error.
So, what could be the better solution?
ASAP, I plan to migrate my system on apache 2.4.x.
In the meantime it would be nice that version 2.2 was corrected.
Best Regards
Michele MAsè
On Thu, Jan 26, 2017 at 1:27 AM, Yann Ylavic <ylavic.dev@xxxxxxxxx> wrote:
Hi,
On Wed, Jan 25, 2017 at 10:33 PM, Michele Mase' <michele.mase@xxxxxxxxx> wrote:
> <?php
I checked restarts with valgrind on latest 2.2.x and found this fixes:
Index: modules/ssl/mod_ssl.c
===================================================================
--- modules/ssl/mod_ssl.c (revision 1778094)
+++ modules/ssl/mod_ssl.c (working copy)
@@ -277,7 +277,12 @@ static apr_status_t ssl_cleanup_pre_config(void *d
/* Don't call ERR_free_strings here; ERR_load_*_strings only
* actually load the error strings once per process due to static
* variable abuse in OpenSSL. */
+#if (OPENSSL_VERSION_NUMBER >= 0x00090805f)
+ ERR_free_strings();
+#endif
+ sk_SSL_COMP_free(SSL_COMP_get_compression_methods());
+
/* Also don't call CRYPTO_cleanup_all_ex_data here; any registered
* ex_data indices may have been cached in static variables in
* OpenSSL; removing them may cause havoc. Notably, with OpenSSL
Index: modules/ssl/ssl_util_ssl.c
===================================================================
--- modules/ssl/ssl_util_ssl.c (revision 1778094)
+++ modules/ssl/ssl_util_ssl.c (working copy)
@@ -311,6 +311,7 @@ BOOL SSL_X509_isSGC(X509 *cert)
break;
}
}
+ EXTENDED_KEY_USAGE_free(sk);
}
}
return is_sgc;
__
> Tomorrow, probably i'll open a ticket with redhat.
The first one (ERR_free_strings) is fixed in 2.4.x and seems to be
backported in redhat's 2.2.* already (at least in
"httpd-2.2.15-56.el6_8.3.src.rpm").
The second one (sk_SSL_COMP_free) is nowhere, neither in httpd nor
redhat (AFAICT) releases.
The last and biggest one (EXTENDED_KEY_USAGE_free) is 2.2 code only
(2.4 not concerned), and seems to affect both httpd-2.x and redhat's.
I won't be able to verify how it affects the different openssl
versions (hence commit anything) in the next few days, just wanted to
notify before being away...
Regards,
Yann.
|
