20:00 < skvidal> meeting? 20:00 * skvidal awaits 20:00 * dgilmore is here 20:00 < dgilmore> mmcgrath: 20:00 < dgilmore> mmcgrath: 20:00 < dgilmore> mmcgrath: 20:00 < mmcgrath> #startmeeting 20:00 < zodbot> Meeting started Thu Dec 10 20:00:40 2009 UTC. The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00 < mmcgrath> #topic Who's here? 20:00 * mmcgrath is 20:00 -!- zodbot changed the topic of #fedora-meeting to: Who's here? 20:00 * dgilmore 20:00 < mmcgrath> smooge: ping 20:00 -!- a-k [n=akistler@2002:638e:1ba0:3:20d:56ff:fe10:bb8d] has joined #fedora-meeting 20:01 < smooge> here 20:01 * a-k is 20:01 < smooge> putting clothes in dryer 20:01 < mmcgrath> So the outage is this weekend, I figured we should all go over the timelines and such 20:01 < mmcgrath> jwb: This might interest you 20:01 < mmcgrath> Oxf13: ^^ 20:02 * skvidal is here 20:02 * jwb is semi here 20:02 < mmcgrath> So here's the basics 20:02 < mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2 20:02 * nirik is sitting in the back in the cheap seats. 20:02 < a-k> mmcgrath: #topic? 20:02 < mmcgrath> #action - Sometime before Sat. Morning mail will start to go through bastion3 in PHX2 20:02 < mmcgrath> #topic Timeline 20:02 -!- zodbot changed the topic of #fedora-meeting to: Timeline 20:02 < smooge> #topic movement to PHX 20:02 < mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2 20:03 < mmcgrath> #action - Sometime before Sat. Morning mail will start to go through bastion3 in PHX2 20:03 * mmcgrath doesn't get this bot stuff sometimes 20:03 < mmcgrath> The mail part is still in an unknown state to me 20:03 < mmcgrath> I've got a ticket open, I've pinged people about it's priority, I'm waiting to hear back. 20:03 < mmcgrath> #action - Smooge is working on some resolvers for us, those should be in place ASAP but isn't a blocker for the move. 20:03 < mmcgrath> it is a blocker to start turning things on though. 20:04 < dgilmore> mmcgrath: do we have the new bastion host up? 20:04 < mmcgrath> dgilmore: yes, bastion3.fedoraproject.org is up and running 20:04 < mmcgrath> Ok, this takes us to Friday morning. 20:04 < smooge> #action - Smooge will email out named scheme for PHX2 resolve domain 20:04 < mmcgrath> #action jwb/ someone in releng will tell me as soon as they are done writing to /mnt 20:05 < mmcgrath> #action at that time I'll mount the public mirrors as read only. 20:05 < jwb> mmcgrath, will likely be two sets of things there 20:05 < jwb> updates, and rawhide 20:05 < mmcgrath> jwb: totally fine, just let me know when it's done. 20:05 < jwb> k 20:05 < mmcgrath> Ok, this is where the coordination stuff happens. 20:05 < mmcgrath> I'll be flying to PHX at noon on Friday. 20:05 < mmcgrath> smooge: when do you leave? 20:06 < smooge> I leave at ~0900 tomorrow morning 20:06 < mmcgrath> that's localtime? 20:06 < dgilmore> smooge: MST 20:06 -!- Sonar_Guy [i=Who@fedora/sonarguy] has quit Read error: 104 (Connection reset by peer) 20:06 < mmcgrath> k 20:07 < mmcgrath> We both have access to PHX1 and PHX2 from Friday to Wed I believe 20:07 * SmootherFrOgZ is around 20:07 < smooge> sorry 0800 tomorrow mornign ariving at ~0900 MST 20:07 < mmcgrath> k 20:07 < mmcgrath> So I'll be heading to PHX1 in the afternoon to give everything a final look over 20:07 < mmcgrath> this is still Friday. 20:08 -!- Sonar_Guy [i=Who@fedora/sonarguy] has joined #fedora-meeting 20:08 < smooge> mmcgrath, when do you get there 20:08 < mmcgrath> I'll probably shut down stuff like db1 since it's not being used at that point. 20:08 < smooge> nm.. see now 20:08 < mmcgrath> I get in at 3:30 20:08 < smooge> 03:30 or 15:30 20:08 < mmcgrath> 15 20:08 < skvidal> what's the plan for mail at this point? 20:09 * mdomsch is here finally 20:09 < mmcgrath> skvidal: I said that already :) 20:09 < skvidal> oh 20:09 < skvidal> it had just scrolled off 20:09 < skvidal> so I missed it on the divider :( 20:09 < skvidal> sorry 20:09 < mmcgrath> skvidal: I've got a ticket open and have pinged people about it's priority 20:09 < mmcgrath> no worries. 20:09 < mmcgrath> I'm hoping for that to get done tonight, if it's not done by then I'm going to need to hand that off to someone to keep bugging people until it gets done because I won't be around on Friday 20:09 < mmcgrath> skvidal: but the good news is we aren't the blockers on that AFAIK. 20:10 < mmcgrath> bastion3 is setup and listening and ready to forward mail. 20:10 < mmcgrath> Ok, so here's what is left 20:10 < mmcgrath> #action Friday night they'll be configuring the new netapp head unit in PHX2 20:10 < mmcgrath> Then sleep 20:10 < mmcgrath> #action Smooge and Mike will meet at PHX1 at 05:15 am 20:11 < mmcgrath> #action Disable nagios 20:11 < mmcgrath> #action make a final backup of sigul and a few other bits (this will probably be done the night before, just forgot to meniton it) 20:11 < mmcgrath> #action power everything down 20:11 < mmcgrath> #action movers will arrive at 05:45 to move 20:11 < jwb> action pray 20:11 < jwb> :) 20:12 < mmcgrath> The movers will actually be helping us unhook cables and such. 20:12 < mmcgrath> oh 20:12 < dgilmore> mmcgrath: buildsys we should disable friday night 20:12 < mmcgrath> #action I'll be taking a sigul drive, and smooge will have the other. 20:12 < jwb> ah, good 20:12 < mmcgrath> dgilmore: we can do it that morning if we want or the night before 20:12 < smooge> mmcgrath, make sure its got both as being bootable. 20:12 < mmcgrath> The movers themselves will actually be loading the entire racks onto the truck. 20:12 < smooge> sorry thats meant more as I need to ... 20:13 < smooge> mmcgrath, they will be? 20:13 < mmcgrath> yep, I have no idea how that'll work but I bet it's going to be wicked awesome. 20:13 < smooge> we aren't getting new racks in PHX or they will move them from one to another there? 20:13 < dgilmore> mmcgrath: lets do it at night make sure all in progress builds complete 20:13 < mmcgrath> dgilmore: sure 20:13 < mmcgrath> #action meet the movers at PHX2 where they'll unload the racks 20:14 < mmcgrath> #action we'll then be moving servers from the old racks (which I guess are now sitting near the new racks) and re-racking everything. 20:14 * mmcgrath will bring bandaids 20:14 < mmcgrath> now this is where part of our story has changed a bit 20:14 < mmcgrath> we were under the impression contractors would be on site to re-wire everything 20:14 < mmcgrath> that's not true 20:14 < pjones> bandaids? consider stopping by autozone and getting some work gloves. 20:15 < mmcgrath> Jonathan (the guy normally onsite at PHX2) is going to be pre-wiring some of the racks. 20:15 < mmcgrath> depending on how far he gets we may have a lot of cabling to do ourselves or none. 20:15 < smooge> pjones, I have had a rack go through teflon armored gloves.. they need blood to work 20:15 < mmcgrath> :) 20:15 < mdomsch> as electronics need smoke 20:15 < pjones> surely you mean kevlar? 20:15 < mmcgrath> so lets see where that leaves us... 20:16 < mmcgrath> #action start powering services back on 20:16 < mmcgrath> let me get the priority list... 20:16 < pjones> (and in any case; knives go through kevlar more easily than they go through canvas. the high velocity of something like a bullet is where kevlar gets its strength from...) 20:16 < mmcgrath> #action get the app servers online first 20:16 < mmcgrath> #action then the buildsystem 20:17 < mmcgrath> I only put them in that order because the app servers should be very straight forward. 20:17 < dgilmore> do we have dhcp? 20:17 < mmcgrath> dgilmore: yeah, and we run it 20:17 -!- sijis_afk [n=sijis@xxxxxxxxxxxx] has joined #fedora-meeting 20:17 < dgilmore> should we reconfigure all the builders first for the new networks 20:17 * mmcgrath is skipping the network config stuff for a later part of the meeting 20:17 < dgilmore> ok 20:17 < mmcgrath> because some of it is quite a bit different 20:17 < mmcgrath> but we'll get to that. 20:17 < mmcgrath> Now it's while we're re-racking this stuff that we will need help from people. 20:18 < mmcgrath> I've already worked on a checklist via CSI 20:18 < mmcgrath> basically I'm looking for people to re-certify these hosts 20:18 < mmcgrath> what does that mean? 20:18 < smooge> pjones, I meant kevlar.. but teflon would expect why I couldn't hold anything 20:18 < mmcgrath> verify remote management, power and cyclades works, as well as network. 20:18 < mmcgrath> this is stuff anyone in sysadmin-main can do. 20:19 < mmcgrath> Ok, so any questions about that? 20:19 < smooge> mmcgrath, pointer to checklist 20:19 < mmcgrath> smooge: I'm going to be adding that info to the ticket in a moment - 20:19 < mmcgrath> https://fedorahosted.org/fedora-infrastructure/ticket/1845 20:19 < smooge> ok cool 20:19 < mmcgrath> Ok, so as far as timeline goes and the physical work of things are there any questions? 20:20 < mmcgrath> Sometime after the move we'll be moving the db hosts back to their normal machines. 20:20 < mmcgrath> Ok, so the next topic is about what the PHX2 world will look like. 20:20 < mmcgrath> #topic PHX2 - what does it look like? 20:20 -!- zodbot changed the topic of #fedora-meeting to: PHX2 - what does it look like? 20:21 < mmcgrath> So in PHX2 we're going to have 3 networks and might add a 4th one later. 20:21 < mmcgrath> The 3 networks are as follows 20:21 -!- rdieter [n=rdieter@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:21 < mmcgrath> 1) public network - 10.5.126 20:21 < mmcgrath> 2) build network - 10.5.125 20:21 < mmcgrath> 2) storage network - 10.5.127 20:21 < mmcgrath> the majority of our services will end up on the buidl network 20:21 < mmcgrath> that should be 3) storage :) 20:22 < mmcgrath> and the majority of our services will end up on the public network, not build. 20:22 * mmcgrath needs more sleep. 20:22 < mmcgrath> the build network will be mostly for releng 20:22 < mmcgrath> and the storage network will be for nfs traffic and possibly backups. 20:22 < mmcgrath> I'm going to be doing tests to find out where backups best work and have the least impact on other things. 20:23 < mmcgrath> This also means that several of our hosts will now have multiple IP addresses. 20:23 -!- notting [n=notting@redhat/notting] has joined #fedora-meeting 20:23 < mmcgrath> for example relengX will be on the build and storage network. 20:23 < mmcgrath> we won't be routing them. 20:23 -!- sandorlev [n=sandorle@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:23 < mmcgrath> Anyone have any questions on this? 20:23 < sandorlev> hello 20:24 < mmcgrath> Ok. so that's really it. 20:24 < mmcgrath> I'm working on a spreadsheet right now that I'd appreciate some second looks at. 20:24 < mdomsch> I presume we'll only have DNS A records for the inbound service IPs on those dual-homed boxes 20:25 < mmcgrath> mdomsch: I believe so. 20:25 < mdomsch> e.g. build1's IP on network 3 won't have an A record 20:25 < mmcgrath> we'll be running our own DNS servers in PHX2, smooge is working on that now. 20:25 < mmcgrath> mdomsch: correct but it will probably have a reverse. 20:25 < mmcgrath> smooge: what do you think? 20:25 < smooge> i am going to have two boxes eventually but at the moment just one.. ns001.phx2.fedoraproject.org 20:26 < mdomsch> mmcgrath, makes sense 20:26 < mdomsch> reverse is good to keep track of allocated addresses at least 20:27 < smooge> the hosts will just be on the phx2 subdomain which will be a 'hidden' domain (not on public dns since it will contain 192 and 10.x ips) 20:27 < mmcgrath> smooge: also on the meeting yesterday I think it was requested that we RH to still be able to do lookups and reverse lookups so we might be stuck doing fedora.phx2.redhat.com 20:27 < mmcgrath> unless there's an easy way to just mask it for them? 20:27 * mmcgrath hasn't thought about that much. 20:27 < smooge> I am going to keep it simple in version one because we don't ahve a lot of time.. but dual homed would be later 20:28 < smooge> mmcgrath, I think in either case they will be able to do zone transfers from our servers 20:28 < mmcgrath> smooge: wise idea 20:28 < mmcgrath> Ok, so any questions about this? 20:28 < smooge> so we could call it fedoraproject.int and it should be good 20:29 < mmcgrath> at present there is no firewall preventing access to or from any of the 3 networks I've listed. 20:29 < mmcgrath> we'll be doing that later, just ran out of time. 20:29 < nb> i would suggest not using .int because .int is a real TLD 20:29 < smooge> nb it was a joke sorry 20:29 -!- nucleo [n=nucleo@fedora/nucleo] has quit Remote closed the connection 20:29 < nb> smooge, oh ok :) 20:29 < mmcgrath> Ok, so anyone else have any questions about this? 20:29 < mmcgrath> I really have no estimate for how long this will take. 20:30 < mmcgrath> I've never moved this many machines before and I have no idea how much is already done and how many people we'll have helping 20:30 < smooge> mmcgrath, you are leaving on Tuesday correct? 20:30 < smooge> smooge will be leaving on Wednesday so we wont lose coverage afterwords 20:30 < smooge> and we have problems like "Oh crap that disk drive didn't make it." 20:30 < mmcgrath> smooge: I'm not sure actually I need to look at my trip info. 20:31 < smooge> in case of real emergency I will just get back in the car and drive back til we are stable. 20:31 < mmcgrath> so that's really all I have to discuss 20:31 < mmcgrath> anyone else have any questions or comments? 20:31 < mmcgrath> if not I'll get back to my IP list and let smooge get back to DNS stuff. 20:32 -!- nucleo [n=nucleo@fedora/nucleo] has joined #fedora-meeting 20:32 < nb> anything non-main people can do to help? 20:32 < mmcgrath> #topic open floor 20:32 -!- zodbot changed the topic of #fedora-meeting to: open floor 20:32 < mmcgrath> nb: I'm hoping the non-main people will help with logistics and troubleshooting 20:32 < nb> ok 20:32 < mmcgrath> when the services start coming back online and interacting with eachother I suspect there will be lots of little bugs 20:32 < mmcgrath> like say, app2's network is behaving poorly 20:33 < mmcgrath> nb: but also keeping watch in #fedora-admin and #fedora 20:33 * stickster wants to say a big "thank you" from all the people in Fedora that won't know to whom they should say it... for working so hard to keep us in business during the move. 20:34 < mmcgrath> nb: but yeah, when stuff comes back online smooge and I will probably say stuff like "the wiki is back" 20:34 < mmcgrath> we'll need verification of that :) 20:34 -!- sandorlev [n=sandorle@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit "Have to go" 20:34 < nb> ok 20:34 < sijis_afk> i know i've missed most of the meeting.. but its this weekend, right? 20:34 < mmcgrath> sijis_afk: correct. 20:34 < mmcgrath> stickster: :) 20:34 < mmcgrath> any other questions? 20:34 < sijis_afk> i should be around and help if possible 20:34 -!- sijis_afk is now known as sijis 20:35 < mmcgrath> when I'm done with this IP list I could use some extra eyes 20:35 * nb will be around saturday and sunday afternoon/evening 20:35 < mmcgrath> Anyone that'll be around in the next hour or so just look, try to find duplicates or where I've done something stupid. 20:35 < nb> mmcgrath, ok 20:35 < mmcgrath> If no one has anything else, I'll close the meeting in 30 20:36 < mmcgrath> alllrighty 20:36 < mmcgrath> #endmeeting 20:36 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel for meeting schedule 20:36 < zodbot> Meeting ended Thu Dec 10 20:36:46 2009 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . 20:36 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting.2009-12-10-20.00.html 20:36 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting.2009-12-10-20.00.txt 20:36 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2009-12-10/fedora-meeting.2009-12-10-20.00.log.html
Attachment:
pgpdhunBkR4rs.pgp
Description: PGP signature
_______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
