Thomas Chung wrote:
On 4/5/07, Thomas Chung <tchung@xxxxxxxxxxxxxxxxx> wrote:
Are we experiencing an issue with our wiki?
I'm seeing following all pages:
Bad Gateway
The proxy server received an invalid response from an upstream server.
Apache/2.0.52 (Red Hat) Server at fedoraproject.org Port 80
It seems the problem has been resolved now. :)
Regards,
This was actually one of our worst crashes. I've got some plans I'm
working on but we'll need the netapps for them to work. Here's a
roundup of what happened.
App2 (where the wiki is actually stored) was having an issue with httpd
starting. I haven't looked into this error (yet) but here's the message:
[Thu Apr 05 11:34:17 2007] [error] (28)No space left on device:
mod_python: Failed to create global mutex 1 of 4 (/tmp/mpmtx309251).
I verified that the box had enough space and inodes (we're talking 70%
free). The box was behaving strangely so I decided to reboot it.
Normally not a problem except this box still had the older iptables
rules on it which causes excessive logging. With connections coming
into it from the proxy servers and app1 the box quickly got
overwhelmed. It locked and I couldn't log in via the console so I
killed the box. When it came back up the grub settings were off so I
could not console into it to see why the boot was hanging (I suspect
fsck). We couldn't PXE that box because its on a different network from
the DHCP server so no rescue mode. Completely lucky for me, I happened
to be here. Stuck a CD in and we were off.
This was bad and could have been much worse. I'm going to take a look
at our options to make sure it doesn't happen again.
-Mike
