Ah finally - problem solved.
After doing some digging, and thanks to much help from Boyle, I
managed to solve this extremely frustrating problem.
As it first appeared, Apache was shutting down at random times, and
then 'waking' up again, only to die again seconds later.
What in fact happened, was that the previous Sys. Admin had set up a
`mon` script, to try and start the webserver if it was down on a
different machine on our LAN (like a load balancer - if server A's
httpd was dead, `mon` would kick in and start httpd on server B).
Anyhow, I remember going through my hosts file last week, and
commenting out some (seemingly) obsolete values. Ofcourse, one of
these was the IP/host for the other load balance server. So what would
happen, is that `mon` would attempt to contact the other server,
couldn't and then try to restart my webserver. Somewhere along the
line this keeps going wrong and kills my Apache.
Here are the responsible lines from the `mon` config file:
------------------
hostgroup ted1 jack_ted
watch ted1
service ping
interval 21m
monitor fping.monitor
allow_empty_group
period wd {Sun-Sat}
alertevery 45m
service http
interval 24m
monitor http.monitor
allow_empty_group
period wd {Sun-Sat}
alert http.restart.action eth0:1
alertevery 40m
-----------------
I hope this will help someone else out there.... because it sure is
frustrating to not know whats going on.
Cheerz,
John
On 6/28/05, AussieJohn <aussiejohn@xxxxxxxxx> wrote:
> On 6/28/05, Boyle Owen <Owen.Boyle@xxxxxxx> wrote:
> > Let's recap: You have a private webserver connected only to a LAN which goes through repeated restart cycles. No cronjobs or daemons or other users appear to be responsible.
> >
>
> definately no cron jobs responsible - I checked every one of them.
>
> > How are you starting apache? Is it, by any chance, started as an internet service (via /etc/inetd.conf)? If so, inetd could be stopping and starting it as required (this is a very bad way to run apache!) Apache should always run as a stand-alone process (do ps -ef | grep httpd and check you get it).
> >
>
> I actually removed apache from starting through inetd a few months ago...
>
> > If not, the only other thing I can think of is to cross-reference the restarts with the access_log using the timestamp and see if there are any requests which coincide. Then try to reproduce the restart by re-issuing those requests.
> >
>
> What I managed to find out, was through looking at /var/log/messages
> that there seemed to be some kind of process throwing lots of errors
> whilst attempting to contact http on a different server on the LAN ..
> As it turns out, the previous sysadmin had set up `mon` ... after
> looking through the mon config file i saw some entries that were
> worrying. I not 100% sure if this was the actual cause, I guess I will
> find out in the morning when I get to the office. If this was indeed
> the cause of my server dying, (and I will try to test this as
> thoroughly as I can), I will post the results here and on my website,
> so that hopefully this won't happen to anyone else as it is terribly
> frustrating.
>
> Cheerz,
>
> John
>
> > Rgds,
> > Owen Boyle
> > Disclaimer: Any disclaimer attached to this message may be ignored.
> >
> > > -----Original Message-----
> > > From: AussieJohn [mailto:aussiejohn@xxxxxxxxx]
> > > Sent: Dienstag, 28. Juni 2005 01:44
> > > To: users@xxxxxxxxxxxxxxxx
> > > Subject: Re: FW: [users@httpd] Apache dies unexpectedly
> > >
> > >
> > > On 6/27/05, AussieJohn <aussiejohn@xxxxxxxxx> wrote:
> > > > On 6/27/05, Boyle Owen <Owen.Boyle@xxxxxxx> wrote:
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: AussieJohn [mailto:aussiejohn@xxxxxxxxx]
> > > > > > Sent: Montag, 27. Juni 2005 13:29
> > > > > >
> > > > > > > > Could it be a CGI program? Does apache start as root?
> > > > > > > >
> > > > > > -----
> > > > > > it could indeed be a cgi (PHP) program - allthough quite
> > > > > > often I am not
> > > > > > using the webserver, and will still die, hence suggesting to
> > > > > > me that it is
> > > > > > not a CGI script causing this.
> > > > >
> > > > > You say "... I am not using the webserver ..."
> > > > >
> > > > > Eh? It is the clients on the WWW who use the webserver -
> > > or do you mean that it is not connected to the internet?
> > > > >
> > > > > To be clear, what I'm wondering is whether someone has
> > > written a script somewhere which is causing a restart when it
> > > gets requested.
> > > > >
> > > > > This is why I asked if apache is started as root. If not,
> > > then a CGI script may be able to issue a restart directly.
> > > Howver, even if apache is started as root, the CGi could
> > > touch a file somewhere and if make were running on the root
> > > crontab with the touch file as a dependency it could make a
> > > restart (this is a common ploy to restart automatically after
> > > some file has been touched).
> > > > >
> > > >
> > > > ---
> > > > Oh, sorry for not clarifying this earlier, but the
> > > webserver runs on my
> > > > development machine, and thus does not get any traffic from
> > > the outside world.
> > > > I have been manually starting apache as root (either when
> > > logged in to a
> > > > root shell, or thru 'sudo').
> > > >
> > > > Though I do remember setting apache to run as its 'own'
> > > user and group
> > > > (apache and htdocs respectively) ... this user and group have access
> > > > to read/write/exec on the webservers document root.
> > > >
> > > > anyway - i may get some sleep - and maybe when i get to
> > > work tomorrow the
> > > > magix tux-fairies will have fixed my problem - if not I'm
> > > sure I will
> > > > reply with some more
> > > > details on the server config and what not...
> > > >
> > > > john
> > > >
> > >
> > > ------------
> > > Ok here a <snip> from my error_log as it tranpired last night (while I
> > > was ofcourse not in the office.)
> > >
> > > [Tue Jun 28 07:04:33 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 07:04:33 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 07:04:33 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 07:04:38 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=14093)
> > > [Tue Jun 28 07:04:38 2005] [notice] caught SIGTERM, shutting down
> > > [Tue Jun 28 07:04:49 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 07:04:49 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 07:04:49 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 07:04:55 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=14135)
> > > [Tue Jun 28 07:04:55 2005] [notice] caught SIGTERM, shutting down
> > > [Tue Jun 28 07:49:50 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 07:49:50 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 07:49:50 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 07:49:55 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=15003)
> > > [Tue Jun 28 07:49:55 2005] [notice] caught SIGTERM, shutting down
> > > [Tue Jun 28 07:50:07 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 07:50:07 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 07:50:07 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 07:50:12 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=15045)
> > > [Tue Jun 28 07:50:12 2005] [notice] caught SIGTERM, shutting down
> > > [Tue Jun 28 08:35:07 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 08:35:07 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 08:35:07 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 08:35:12 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=15919)
> > > [Tue Jun 28 08:35:12 2005] [notice] caught SIGTERM, shutting down
> > > [Tue Jun 28 08:35:24 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > > configured -- resuming normal operations
> > > [Tue Jun 28 08:35:24 2005] [info] Server built: Apr 18 2005 14:28:22
> > > [Tue Jun 28 08:35:24 2005] [debug] prefork.c(956): AcceptMutex:
> > > sysvsem (default: sysvsem)
> > > [Tue Jun 28 08:35:29 2005] [info] removed PID file
> > > /usr/local/apache2/logs/httpd.pid (pid=15961)
> > > [Tue Jun 28 08:35:29 2005] [notice] caught SIGTERM, shutting down
> > >
> > > apologies for the length, but i thought it might be helpful
> > > for you to see.
> > >
> > > I just did a `ps x` and saw that 'fam' was running ... could fam be
> > > set to monitor my PID file ?
> > > and then remove it ?
> > >
> > > Also - the other file in /etc/init.d was 'httpd.rpmsave' - which I've
> > > moved to /tmp for now ...
> > > I also edited the 'httpd' script in /etc/init.d/ as I noticed that the
> > > PID file location was set to be in /var/run (but my actual PID file is
> > > in /usr/local/apache2/logs/
> > >
> > > I will keep you updated as I go with this ...
> > >
> > > john
> > > -------------
> > >
> > > >
> > > > ---
> > > > > Rgds,
> > > > > Owen Boyle
> > > > > Disclaimer: Any disclaimer attached to this message may
> > > be ignored.
> > > > >
> > > > >
> > > > > >
> > > > > > also what might be interesting to note, is that I did to a
> > > > > > reboot of my system,
> > > > > > as I figured this may clear out zombified processes and
> > > other nasty
> > > > > > things that may have ended up in memory/in apache thread.
> > > > > >
> > > > > > Alas, to no avail, approximately 10/15 mins after reboot the
> > > > > > server died again.
> > > > > >
> > > > > > -----
> > > > > >
> > > > > > > > Rgds,
> > > > > > > > Owen Boyle
> > > > > > > > Disclaimer: Any disclaimer attached to this message may
> > > > > > be ignored.
> > > > > > > >
> > > > > > > >
> > > > > > > > > This is extremely odd, as the webserver
> > > > > > > > > ofcourse doesn't
> > > > > > > > > get used on the weekend as it is my workstation (I'm a web
> > > > > > > > developer).
> > > > > > > > > the only thing I could think of was a misbehaving profiler
> > > > > > > > or module.
> > > > > > > > > The only things I added before the server started
> > > dying was a
> > > > > > > > > virtualhost
> > > > > > > > > and mod_auth.
> > > > > > > > > I have tested with disabling the vhost, mod_auth and
> > > > > > > > > Xdebug (PHP profiler/debugger).
> > > > > > > > >
> > > > > > > > > None of these seem to give me a solution. So I am still
> > > > > > stuck with a
> > > > > > > > > sporadically
> > > > > > > > > dying webserver.
> > > > > > > > > When I get back to work in the morning I will however make
> > > > > > > > sure that
> > > > > > > > > there are no crontabs set. I am 99.9 % sure there
> > > > > > aren;t as I am the
> > > > > > > > > only one that
> > > > > > > > > uses my machine.
> > > > > > > > >
> > > > > > > > > Would their be any other way to find out what
> > > exactly happens
> > > > > > > > > that would kill
> > > > > > > > > the server ?
> > > > > > > > > -------------
> > > > > > > > > > Rgds,
> > > > > > > > > > Owen Boyle
> > > > > > > > > > Disclaimer: Any disclaimer attached to this message may
> > > > > > > > be ignored.
> > > > > > > > > >
> > > > > > > > > <snip>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > "If you really want something in this life, you have to
> > > > > > work for it.
> > > > > > > > > Now, quiet! They're about to announce the lottery
> > > numbers..."
> > > > > > > > > - Homer Simpson
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This message is for the named person's use only. It may
> > > > > > contain confidential, proprietary or legally privileged
> > > > > > information. No confidentiality or privilege is waived or
> > > > > > lost by any mistransmission. If you receive this message in
> > > > > > error, please notify the sender urgently and then immediately
> > > > > > delete the message and any copies of it from your system.
> > > > > > Please also immediately destroy any hardcopies of the
> > > > > > message. You must not, directly or indirectly, use, disclose,
> > > > > > distribute, print, or copy any part of this message if you
> > > > > > are not the intended recipient. The sender's company reserves
> > > > > > the right to monitor all e-mail communications through their
> > > > > > networks. Any views expressed in this message are those of
> > > > > > the individual sender, except where the message states
> > > > > > otherwise and the sender is authorised to state them to be
> > > > > > the views of the sender's company.
> > > > > > >
> > > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > The official User-To-User support forum of the Apache HTTP
> > > > > > Server Project.
> > > > > > > See <URL:http://httpd.apache.org/userslist.html> for
> > > more info.
> > > > > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > > > " from the digest:
> > > users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > "If you really want something in this life, you have to
> > > work for it.
> > > > > > Now, quiet! They're about to announce the lottery numbers..."
> > > > > > - Homer Simpson
> > > > > >
> > > > > >
> > > ---------------------------------------------------------------------
> > > > > > The official User-To-User support forum of the Apache HTTP
> > > > > > Server Project.
> > > > > > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > > > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > > " from the digest:
> > > users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > ---------------------------------------------------------------------
> > > > > The official User-To-User support forum of the Apache
> > > HTTP Server Project.
> > > > > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > "If you really want something in this life, you have to work for it.
> > > > Now, quiet! They're about to announce the lottery numbers..."
> > > > - Homer Simpson
> > > >
> > >
> > >
> > > --
> > > "If you really want something in this life, you have to work for it.
> > > Now, quiet! They're about to announce the lottery numbers..."
> > > - Homer Simpson
> > >
> > > ---------------------------------------------------------------------
> > > The official User-To-User support forum of the Apache HTTP
> > > Server Project.
> > > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP Server Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> >
> >
>
>
> --
> "If you really want something in this life, you have to work for it.
> Now, quiet! They're about to announce the lottery numbers..."
> - Homer Simpson
>
--
"If you really want something in this life, you have to work for it.
Now, quiet! They're about to announce the lottery numbers..."
- Homer Simpson
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
" from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|