Karsten Wade wrote:
On Sun, 2007-05-27 at 17:41 +0100, Damian Myerscough wrote:
Yea, it would. However mod_easvie is able to detect if users a
continuously hitting refresh
to consume bandwidth.
Would it make sense to have a few tricks ready to go? Untested or
unproven items to pull out in response to what happens. Sort of like
what Egon Spengler might pull out in Ghostbusters ...
I know, it could be *worse* than whatever happens, but there is a slim
chance we'll survive.
Actually we've got lots of plans as it is. I've been meaning to send
this out but have been busy so here goes (all, comment and become
familiar as necessary, the more people that know what is going on, the
better)
This is only things related to the F7 launch:
Critical:
1) The static pages
2) The wiki
3) Mirror manager
4) Mirrors (which should be taken care of and synced long before
the actual bit flip)
Presently the static pages are fully redundant and the wiki and mirror
manager are redundant up to their data layers. (netapp and database
respectively)
Plan A)
We currently have 2 proxy servers and 4 application servers in
place. 2 app servers are FC6, 2 are RHEL. At present the two RHEL
boxes load balance the wiki (moin), the two app servers are for mirror
manager (TurboGears). The static pages are on the actual proxy
servers. On release day we'll track the load trend on these 6
machines. As long as release related content is being served, we hold.
Plan B)
If any one of the above pieces begins to fail we can add capacity as
required as well as limiting traffic with iptables. FC6 servers run
mirror manager. RHEL servers run the wiki (though the wiki could
technically be run on FC6 boxes, it just hasn't been done yet, no
reason) and the static pages are run directly on the proxy servers.
Adding application servers is easy, just kick start the box, have it
connect to puppet. Proxy servers are trickier. They exist on a
different network, but the same principal applies. We just contact the
network guys to switch a port to the correct vlan. I've got xen6 so it
has one nic on the app network and one nic on the proxy network.
Plan C)
Start redirecting all traffic to the mirrors (those that have our
web content). The theory is the most efficient transaction our web
servers can do is to take a get and re-direct to a different server.
The big 'gotcha' here is mirror manager. Mirror manager won't know
which mirrors have Fedora 7 until they have it. So we'll have to serve
the public server list at PHX. We're featuring the torrents a bit more
this time compared to last time. Hopefully that will help keep load on
the mirrors lighter, there's little way for us to track this AFAIK.
Unfortunately we don't have full trends from last year because a
different team was running fedora.redhat.com during the last release
(and f.rh.c doesn't even exist this release) We'll get a much better
idea of what we're facing on release day.
In general I'd like to encourage that we hold this configuration unless
there's any simple upgrades or obvious flaws. We're not really in an
'infrastructure change freeze' but I'd encourage everyone needing to
make changes to wait until a day or so after the release. Exceptions,
of course, would include something like bodhi which is of a high
priority. I'll be flying to Germany for Linux Tag tomorrow (Monday) and
will remain there for the release. I'll probably be on German time but
will try to be around as needed or if any problems arise. My cell phone
may be limited though.
Comments?
-Mike
