Apache abort and semaphore leak

["Manas Garg" <manasgarg@xxxxxxxxx>]

Hello,

We had a problem in one of our server where Apache was not coming up properly. The problem got resolved by reboot.

We are running the following version of apache:

sh-3.00# ../bin/httpd -v
Server version: Apache/2.2.2
Server built:   Aug 18 2006 11:42:41

It is running on the following system:

sh-3.00# uname -a
Linux fltwd-intdep01.nevisnetworks.com 2.6.17-1.2142_FC4smp #1 SMP Sat Aug 12 08:16:08 EDT 2006 i686 i686 i386 GNU/Linux

This system has 8 GB of RAM and 2 dual core CPUs.

The load on the system is moderate. When the problem occurred, the services were being started on that system. So, load was practically zero.

We run a watchdog to start Apache if it is not running. Yesterday, we saw that Apache was not coming up and the error log was full of the following messages:

[Tue Mar 18 18:41:50 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:50 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:50 2008] [notice] Digest: generating secret for digest authentication ...
[Tue Mar 18 18:41:50 2008] [notice] Digest: done
[Tue Mar 18 18:41:51 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:51 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:41:51 2008] [warn] pid file /usr/local/software/platform/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apa
che run?
[Tue Mar 18 18:41:51 2008] [error] (13)Permission denied: could not create /usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:41:51 2008] [error] httpd: could not log pid to file /usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:42:51 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:51 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:51 2008] [notice] Digest: generating secret for digest authentication ...
[Tue Mar 18 18:42:51 2008] [notice] Digest: done
[Tue Mar 18 18:42:52 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:52 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Tue Mar 18 18:42:52 2008] [warn] pid file /usr/local/software/platform/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apa
che run?
[Tue Mar 18 18:42:52 2008] [error] (13)Permission denied: could not create /usr/local/software/platform/apache/logs/httpd.pid
[Tue Mar 18 18:42:52 2008] [error] httpd: could not log pid to file /usr/local/software/platform/apache/logs/httpd.pid


We checked the permissions on httpd.pid file and it was writable by Apache. After some time, the error logs become:

[Wed Mar 19 18:48:06 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:48:06 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:48:06 2008] [crit] (28)No space left on device: mod_jk: could not create jk_log_lock
Configuration Failed
[Wed Mar 19 18:49:03 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:49:03 2008] [warn] RSA server certificate CommonName (CN) `fltwd-intdep03.nevisnetworks.com' does NOT match server name!?
[Wed Mar 19 18:49:03 2008] [crit] (28)No space left on device: mod_jk: could not create jk_log_lock
Configuration Failed

We found that the system had run out of the semaphore. Previously, whenever apache was exiting, it was leaving some semaphores in the system. Eventually, the system ran out of semaphores and Apache started giving these errors.

We decided to reboot the machine and after reboot, everything worked fine. So, before reboot, Apache complained about permissions on httpd.pid but after reboot, it didn't. And we didn't change the permissions meanwhile. And anyways, the permissions were correct to begin with.

I tried reproducing this error by stopping Apache, creating httpd.pid as root without any rights to Apache and then starting apache. In this case, Apache didn't print anything in the logs and it didn't leave any semaphores. It just exited silently. So, I feel that the issue that occurred yesterday was not really about file permissions. But it was something else.

I have attached the conf file with this mail. Please let me know if I need to provide more information on this issue.

Thanks & Regards,
Manas Garg

Attachment: httpd.conf
Description: Binary data

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx