Re: FW: [users@httpd] Apache dies unexpectedly

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 6/28/05, Boyle Owen <Owen.Boyle@xxxxxxx> wrote:
> Let's recap: You have a private webserver connected only to a LAN which goes through repeated restart cycles. No cronjobs or daemons or other users appear to be responsible.
> 

definately no cron jobs responsible - I checked every one of them.

> How are you starting apache? Is it, by any chance, started as an internet service (via /etc/inetd.conf)? If so, inetd could be stopping and starting it as required (this is a very bad way to run apache!) Apache should always run as a stand-alone process (do ps -ef | grep httpd and check you get it).
> 

I actually removed apache from starting through inetd a few months ago...

> If not, the only other thing I can think of is to cross-reference the restarts with the access_log using the timestamp and see if there are any requests which coincide. Then try to reproduce the restart by re-issuing those requests.
> 

What I managed to find out, was through looking at /var/log/messages
that there seemed to be some kind of process throwing lots of errors
whilst attempting to contact http on a different server on the LAN ..
As it turns out, the previous sysadmin had set up `mon` ... after
looking through the mon config file i saw some entries that were
worrying. I not 100% sure if this was the actual cause, I guess I will
find out in the morning when I get to the office. If this was indeed
the cause of my server dying, (and I will try to test this as
thoroughly as I can), I will post the results here and on my website,
so that hopefully this won't happen to anyone else as it is terribly
frustrating.

Cheerz, 

John

> Rgds,
> Owen Boyle
> Disclaimer: Any disclaimer attached to this message may be ignored.
> 
> > -----Original Message-----
> > From: AussieJohn [mailto:aussiejohn@xxxxxxxxx]
> > Sent: Dienstag, 28. Juni 2005 01:44
> > To: users@xxxxxxxxxxxxxxxx
> > Subject: Re: FW: [users@httpd] Apache dies unexpectedly
> >
> >
> > On 6/27/05, AussieJohn <aussiejohn@xxxxxxxxx> wrote:
> > > On 6/27/05, Boyle Owen <Owen.Boyle@xxxxxxx> wrote:
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: AussieJohn [mailto:aussiejohn@xxxxxxxxx]
> > > > > Sent: Montag, 27. Juni 2005 13:29
> > > > >
> > > > > > > Could it be a CGI program? Does apache start as root?
> > > > > > >
> > > > > -----
> > > > > it could indeed be a cgi (PHP) program - allthough quite
> > > > > often I am not
> > > > > using the webserver, and will still die, hence suggesting to
> > > > > me that it is
> > > > > not a CGI script causing this.
> > > >
> > > > You say "... I am not using the webserver ..."
> > > >
> > > > Eh? It is the clients on the WWW who use the webserver -
> > or do you mean that it is not connected to the internet?
> > > >
> > > > To be clear, what I'm wondering is whether someone has
> > written a script somewhere which is causing a restart when it
> > gets requested.
> > > >
> > > > This is why I asked if apache is started as root. If not,
> > then a CGI script may be able to issue a restart directly.
> > Howver, even if apache is started as root, the CGi could
> > touch a file somewhere and if make were running on the root
> > crontab with the touch file as a dependency it could make a
> > restart (this is a common ploy to restart automatically after
> > some file has been touched).
> > > >
> > >
> > > ---
> > > Oh, sorry for not clarifying this earlier, but the
> > webserver runs on my
> > > development machine, and thus does not get any traffic from
> > the outside world.
> > > I have been manually starting apache as root (either when
> > logged in to a
> > > root shell, or thru 'sudo').
> > >
> > > Though I do remember setting apache to run as its 'own'
> > user and group
> > > (apache and htdocs respectively) ... this user and group have access
> > > to read/write/exec on the webservers document root.
> > >
> > > anyway - i may get some sleep - and maybe when i get to
> > work tomorrow the
> > > magix tux-fairies will have fixed my problem - if not I'm
> > sure I will
> > > reply with some more
> > > details on the server config and what not...
> > >
> > > john
> > >
> >
> > ------------
> > Ok here a <snip> from my error_log as it tranpired last night (while I
> > was ofcourse not in the office.)
> >
> > [Tue Jun 28 07:04:33 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 07:04:33 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 07:04:33 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 07:04:38 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=14093)
> > [Tue Jun 28 07:04:38 2005] [notice] caught SIGTERM, shutting down
> > [Tue Jun 28 07:04:49 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 07:04:49 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 07:04:49 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 07:04:55 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=14135)
> > [Tue Jun 28 07:04:55 2005] [notice] caught SIGTERM, shutting down
> > [Tue Jun 28 07:49:50 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 07:49:50 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 07:49:50 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 07:49:55 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=15003)
> > [Tue Jun 28 07:49:55 2005] [notice] caught SIGTERM, shutting down
> > [Tue Jun 28 07:50:07 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 07:50:07 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 07:50:07 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 07:50:12 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=15045)
> > [Tue Jun 28 07:50:12 2005] [notice] caught SIGTERM, shutting down
> > [Tue Jun 28 08:35:07 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 08:35:07 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 08:35:07 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 08:35:12 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=15919)
> > [Tue Jun 28 08:35:12 2005] [notice] caught SIGTERM, shutting down
> > [Tue Jun 28 08:35:24 2005] [notice] Apache/2.0.54 (Unix) PHP/4.3.11
> > configured -- resuming normal operations
> > [Tue Jun 28 08:35:24 2005] [info] Server built: Apr 18 2005 14:28:22
> > [Tue Jun 28 08:35:24 2005] [debug] prefork.c(956): AcceptMutex:
> > sysvsem (default: sysvsem)
> > [Tue Jun 28 08:35:29 2005] [info] removed PID file
> > /usr/local/apache2/logs/httpd.pid (pid=15961)
> > [Tue Jun 28 08:35:29 2005] [notice] caught SIGTERM, shutting down
> >
> > apologies for the length, but i thought it might be helpful
> > for you to see.
> >
> > I just did a `ps x` and saw that 'fam' was running ... could fam be
> > set to monitor my PID file ?
> > and then remove it ?
> >
> > Also - the other file in /etc/init.d was 'httpd.rpmsave' - which I've
> > moved to /tmp for now ...
> > I also edited the 'httpd' script in /etc/init.d/ as I noticed that the
> > PID file location was set to be in /var/run (but my actual PID file is
> > in /usr/local/apache2/logs/
> >
> > I will keep you updated as I go with this ...
> >
> > john
> > -------------
> >
> > >
> > > ---
> > > > Rgds,
> > > > Owen Boyle
> > > > Disclaimer: Any disclaimer attached to this message may
> > be ignored.
> > > >
> > > >
> > > > >
> > > > > also what might be interesting to note, is that I did to a
> > > > > reboot of my system,
> > > > > as I figured this may clear out zombified processes and
> > other nasty
> > > > > things that may have ended up in memory/in apache thread.
> > > > >
> > > > > Alas, to no avail, approximately 10/15 mins after reboot the
> > > > > server died again.
> > > > >
> > > > > -----
> > > > >
> > > > > > > Rgds,
> > > > > > > Owen Boyle
> > > > > > > Disclaimer: Any disclaimer attached to this message may
> > > > > be ignored.
> > > > > > >
> > > > > > >
> > > > > > > > This is extremely odd, as the webserver
> > > > > > > > ofcourse doesn't
> > > > > > > > get used on the weekend as it is my workstation (I'm a web
> > > > > > > developer).
> > > > > > > > the only thing I could think of was a misbehaving profiler
> > > > > > > or module.
> > > > > > > > The only things I added before the server started
> > dying was a
> > > > > > > > virtualhost
> > > > > > > > and mod_auth.
> > > > > > > > I have tested with disabling the vhost, mod_auth and
> > > > > > > > Xdebug (PHP profiler/debugger).
> > > > > > > >
> > > > > > > > None of these seem to give me a solution. So I am still
> > > > > stuck with a
> > > > > > > > sporadically
> > > > > > > > dying webserver.
> > > > > > > > When I get back to work in the morning I will however make
> > > > > > > sure that
> > > > > > > > there are no crontabs set. I am 99.9 % sure there
> > > > > aren;t as I am the
> > > > > > > > only one that
> > > > > > > > uses my machine.
> > > > > > > >
> > > > > > > > Would their be any other way to find out what
> > exactly happens
> > > > > > > > that would kill
> > > > > > > > the server ?
> > > > > > > > -------------
> > > > > > > > > Rgds,
> > > > > > > > > Owen Boyle
> > > > > > > > > Disclaimer: Any disclaimer attached to this message may
> > > > > > > be ignored.
> > > > > > > > >
> > > > > > > > <snip>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > "If you really want something in this life, you have to
> > > > > work for it.
> > > > > > > > Now, quiet! They're about to announce the lottery
> > numbers..."
> > > > > > > > - Homer Simpson
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > This message is for the named person's use only. It may
> > > > > contain confidential, proprietary or legally privileged
> > > > > information. No confidentiality or privilege is waived or
> > > > > lost by any mistransmission. If you receive this message in
> > > > > error, please notify the sender urgently and then immediately
> > > > > delete the message and any copies of it from your system.
> > > > > Please also immediately destroy any hardcopies of the
> > > > > message. You must not, directly or indirectly, use, disclose,
> > > > > distribute, print, or copy any part of this message if you
> > > > > are not the intended recipient. The sender's company reserves
> > > > > the right to monitor all e-mail communications through their
> > > > > networks. Any views expressed in this message are those of
> > > > > the individual sender, except where the message states
> > > > > otherwise and the sender is authorised to state them to be
> > > > > the views of the sender's company.
> > > > > >
> > > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > > The official User-To-User support forum of the Apache HTTP
> > > > > Server Project.
> > > > > > See <URL:http://httpd.apache.org/userslist.html> for
> > more info.
> > > > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > >    "   from the digest:
> > users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > "If you really want something in this life, you have to
> > work for it.
> > > > > Now, quiet! They're about to announce the lottery numbers..."
> > > > > - Homer Simpson
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > The official User-To-User support forum of the Apache HTTP
> > > > > Server Project.
> > > > > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > > >    "   from the digest:
> > users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > > >
> > > > >
> > > >
> > > >
> > ---------------------------------------------------------------------
> > > > The official User-To-User support forum of the Apache
> > HTTP Server Project.
> > > > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > > > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> > > >    "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > > > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> > > >
> > > >
> > >
> > >
> > > --
> > > "If you really want something in this life, you have to work for it.
> > > Now, quiet! They're about to announce the lottery numbers..."
> > > - Homer Simpson
> > >
> >
> >
> > --
> > "If you really want something in this life, you have to work for it.
> > Now, quiet! They're about to announce the lottery numbers..."
> > - Homer Simpson
> >
> > ---------------------------------------------------------------------
> > The official User-To-User support forum of the Apache HTTP
> > Server Project.
> > See <URL:http://httpd.apache.org/userslist.html> for more info.
> > To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
> >    "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> > For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> >
> >
> 
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
>    "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
> For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
> 
> 


-- 
"If you really want something in this life, you have to work for it.
Now, quiet! They're about to announce the lottery numbers..."
- Homer Simpson

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx
   "   from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx



[Index of Archives]     [Open SSH Users]     [Linux ACPI]     [Linux Kernel]     [Linux Laptop]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Squid]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Video 4 Linux]     [Device Mapper]

  Powered by Linux