Update: On 2008-07-23 14:56 Henrik Levkowetz said the following:
www3.tools.ietf.org is currently very heavily loaded, and it's not possible to ssh in to it. I've removed the A and AAAA records for it from the zone file for tools.ietf.org, so any long delays you may have experienced reaching the wg status pages, RFCs and drafts on tools.ietf.org should be eliminated. The wiki pages and trackers are normally only available through www3, though, unless I switch over to another machine as master. I'm going to try to get www3 back to normal a bit longer, before I take that step, though. Sorry for the temporary unavailability of those services.
www3.tools.ietf.org is up and running again. What triggered the incident is that during a short period of time yesterday the server was hit by about 14000 requests from a single client. This by itself should have made it sluggish, but shouldn't have made it unresponsive. However, the configuration of the apache server permitted a too high number of server processes, which led to the machine running out of available RAM first, and later running out of swap memory. In the end, a power-down/power-up was necessary to bring it back. I've now changed the apache configuration so that the maximum number of server processes will be a number I've seen the server survive at its earlier highest load level, which should prevent this particular situation from arising again. However, over the weeks after the IETF I'm going to add some more fail-over mechanisms to ensure that services (like Wikis and Issue Trackers) which take input from users, and thus can't easily be served simultaneously from multiple servers, will be be automatically migrated to another server if the master server for those services fails. Hopefully that will mean that the next time a tools server has problems (and of course that will happen again, at some point) nobody will notice it except the administrator ;-) Best, Henrik _______________________________________________ Ietf@xxxxxxxx https://www.ietf.org/mailman/listinfo/ietf