Strange Apache error (appears to be related to bad pidfile)

[Chris Brooks <madbury@xxxxxxxxx>]

Good morning,

I've had a strange problem with Apache twice in the past week, and I'm out of ideas as to what might be going on.  


Here's the background: I run a website that uses a plain-vanilla
Apache server to serve images, and uses mod_proxy to forward requests
for everything else to a backend mod_perl server on a different port on
the same ip address.  



The problem that I ran into is that after running on the same
configuration for years, and without a reboot for a month or so, the
plain-vanilla Apache server  stopped serving requests.

• I did not intentionally make any changes that I would expect to result in problems, either to file permissions or to the userid that Apache runs under.
  
• My first try was to execute apachectl stop and then apachectl start.  apachectl printed: "/usr/local/apache-light/bin/apachectl start: httpd started".  However, it wasn't running (which I verified by trying to run apachectl status and then apachectl stop).
•  The first time this happened, I was running Apache 1.3.37.  I downloaded the tarball for Apache 1.3.41 and installed that in place of 1.3.37.  The second time this error occurred it occurred with 1.3.41.
• No messages were written to the access_log or error_log.  (I have since been told that I should interact directly with httpd, and that this would likely result in messages written to the log.)
  
• /usr/local/apache-light/bin/apachectl configtest returned "Syntax OK"
• apachectl points to the correct pidfile (at least it's the one specified in /usr/local/apache-light/conf/httpd.con), and it looks to me like root has perfectly reasonable rights to the pidfile.

I was eventually able to restart the apache process the first time by
deleting the pidfile.  When I ran apachectl start without a pidfile,
the apache process started up and ran without problems for 3 days.  The
second time this happened, it wasn't enough to delete the pid file.  I
again reinstalled apache, and went so far as to reboot the server. 
(fsck did not find any file system errors on reboot.)  This time, I was only able
to restart the sever after running the following:

[root@data2]# touch /usr/local/apache-light/logs/

httpd.pid
[root@data2]# rm -f /usr/local/apache-light/logs/httpd.pid

[root@data2]# /usr/local/apache-light/bin/apachectl start

This finally worked, and the server has been serving requests for 3 days now.

Any ideas on what might be going on?  I'd love to have some ideas before this happens again -- it's been a stressful 45 minutes each time while I scrambled to try to figure out how to restart it.

Thanks!
-Chris