vFAD report - 2011-11-16
Had a number of folks in and out during the FAD and got a lot of things
hashed out.
Tenative Deadlines/timeline:
2011-11-16 - finish creating collab03/04 instances.
2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
2011-11-18 - ask people to test against test instance.
2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
(announce migration and flag day for the 7th announce announce list.)
2011-11-28 - migrate to collab03/04 (evening) (fedorahosted lists move now)
2011-12-07 - flag day to migrate hosted
url's change from fedorahosted.org/git/whatever -> project.fedorahosted.org
2011-12-11 - have to be done with serverbeach01-05
Instances:
collab03/04 to be made later today
rhel6 test hosted to be made tomorrow and data synced to it.
rhel6 or rhel5 instances to be made later once decision is made.
Decisions/plans:
0. lists move to collab
1. projects get cnames in dns.
2. Number and location of new hosted instances up in the air until rhel6 decision is made.
SOP for moving a project:
- copy data to new machine. scm/trac/releases
- create dns cnames for project: foo.fedorahosted.org pointing to the machine they are going to.
- configure apache, migrate trac
- Notify user they need to adjust scm setting to foo.fedorahosted.org/SCM/
create new fedorahosted-announce@xxxxxxxxxxxxxxxxxxxxxx list, invite all current admins,
announce out to users when we announce migration.
==============================================
#fedora-fad: Virtual FedoraHosted activity day
==============================================
Meeting started by nirik at 20:07:07 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-fad/2011-11-16/vfad-fedorahosted.2011-11-16-20.07.log.html
.
Meeting summary
---------------
* Background / Where we are not (nirik, 20:19:52)
* Background / Where we are now (nirik, 20:19:54)
* Planning/Brainstorming (nirik, 20:25:54)
* we need to be migrated off the old machines by 2011-12-11. (nirik,
20:28:47)
* LINK: http://trac.edgewall.org/wiki/0.12/TracUpgrade (nirik,
21:10:54)
* setup rhel6 box to test migrations with in the next few days.
(nirik, 21:19:43)
* evaluate early next week if it's going to be smooth enough or not.
(nirik, 21:19:59)
* setup collab03/04 later today (nirik, 21:20:32)
* 2011-11-16 - finish creating collab03/04 instances. (nirik,
22:04:58)
* 2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
(nirik, 22:04:58)
* 2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
(nirik, 22:04:58)
* (announce migration and flag day for the 7th announce announce
list.) (nirik, 22:04:58)
* 2011-11-28 - test migrate some hosted instances/opt in (nirik,
22:04:59)
* 2011-11-28 - migrate to collab03/04 (evening) (nirik, 22:05:01)
* 2011-12-07 - flag day to migrate hosted (nirik, 22:05:03)
* url's change from fedorahosted.org/git/whatever ->
project.fedorahosted.org (nirik, 22:05:05)
* 2011-12-11 - have to be done with serverbeach01-05 (nirik,
22:05:07)
Meeting ended at 22:52:05 UTC.
Action Items
------------
Action Items, by person
-----------------------
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (206)
* skvidal (110)
* abadger1999 (56)
* j_dulaney (36)
* nokia3510 (30)
* StylusEater (15)
* CodeBlock (12)
* smooge (7)
* zodbot (4)
* athmane (1)
* lmacken (0)
--
20:07:07 <nirik> #startmeeting Virtual FedoraHosted activity day
20:07:07 <zodbot> Meeting started Wed Nov 16 20:07:07 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:07:07 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:07:24 <nirik> #meetingname vfad-fedorahosted
20:07:24 <zodbot> The meeting name has been set to 'vfad-fedorahosted'
20:07:52 <nirik> #chair skvidal abadger1999 smooge lmacken j_dulaney nokia3510
20:07:52 <zodbot> Current chairs: abadger1999 j_dulaney lmacken nirik nokia3510 skvidal smooge
20:08:07 <nirik> who all is around today for the record? ;)
20:08:28 * CodeBlock here
20:08:44 * athmane here
20:08:46 * j_dulaney is eating bacon
20:08:51 * CodeBlock is jealous
20:08:53 <smooge> I am here
20:10:43 <nirik> so, google+ hangout in addition to irc? any no votes?
20:11:11 * nirik is happy to accomodate most folks, but if we can all use it, might help with BW
20:11:15 * j_dulaney is there, too
20:11:31 * CodeBlock wonders if it will let him listen without a mic from work..
20:11:55 <nirik> yeah, you can mute
20:12:23 * nirik installs the 60 packages the "64bit" plugin wants. ;)
20:12:33 <j_dulaney> Oy, lack of mike would be an issue
20:12:48 <j_dulaney> Hmm, maybe I can use the one on my phone
20:13:02 * j_dulaney investigates this possibility
20:13:51 <nirik> yes, the mobile app can join them now too I think.
20:14:17 * abadger1999 hereish
20:14:18 <skvidal> yes it can
20:14:37 * nokia3510 doesn't find much of Fedora on G+ o_0
20:15:10 <skvidal> nokia3510: lots of fedora people
20:15:16 <skvidal> nirik: are you starting the hangout?
20:15:18 <j_dulaney> Are you talking about google talk, specifically?
20:15:30 <nirik> I can... just a sec.
20:15:36 <nokia3510> skvidal: fedorahosted I meant
20:16:07 <skvidal> nokia3510: umm, ok
20:17:59 <nirik> it's acting wonky.
20:18:09 <skvidal> nirik: I can see the hangout - but I can't see anything except for your dog
20:18:09 <nirik> I started it, saw skvidal join, but now it says no one
20:18:12 <skvidal> and I can hear nothing
20:18:29 <skvidal> it appears to be gone now
20:19:01 <nirik> oh well, tried an invite.
20:19:09 <nirik> anyhow, I guess we should just drive on without.
20:19:15 <nirik> and perfect it for next time.
20:19:17 * skvidal starts one
20:19:21 * j_dulaney is installing
20:19:36 * CodeBlock just installed on his workstation at work.
20:19:45 <nirik> I can give a bit of background while folks poke at that.
20:19:52 <nirik> #topic Background / Where we are not
20:19:54 <nirik> #topic Background / Where we are now
20:20:05 <CodeBlock> hi there skvidal
20:20:26 <smooge> oh wait
20:20:53 * CodeBlock chuckles at skvidal abusing his keyboard :)
20:21:17 <nirik> So, we currently have fedorahosted.org.
20:21:20 <nirik> it's rhel5.
20:21:44 <nirik> it's also 1 machine.
20:22:00 <nirik> it handles trac, mailing lists, scm commits and reads.
20:22:07 <skvidal> yah
20:22:14 <skvidal> so - google hangouts appear to be goofy
20:22:33 <nirik> it's a bit less heavily loaded than it was, but it's still a single machine and doing a lot.
20:22:40 <nirik> yeah, mine just crashed firefox. ;(
20:22:52 <StylusEater> skvidal: attempting to install plugin on rhel6 workstation
20:23:12 <nirik> we also have a hosted02 instance that is a rsync copy of hosted01. For disaster recovery and also for backups.
20:23:14 <smooge> I kind of have it working from EL6y
20:23:32 <skvidal> nirik: for hopeful disaster recovery
20:23:46 <skvidal> and there goes chromium
20:23:52 * skvidal gives up on that idea
20:24:16 <nirik> yeah, just too buggy. It was much more stable a while back... perhaps it hates new firefox.
20:24:18 <StylusEater> smooge: so that's what you look like ... all business like
20:24:34 * CodeBlock closes the window... smooge I hear no audio from you btw
20:24:39 <CodeBlock> or very quiet audio maybe...
20:24:57 <StylusEater> smooge: I can hear you.
20:25:03 <CodeBlock> hm.
20:25:07 * CodeBlock heard skvidal ...
20:25:10 <CodeBlock> but not smooge
20:25:12 <nirik> so, what I would like to do today is come up with our plans for migrating this hosted01 instance to rhel6 and also do anything we can easily do to help out to make it more distributed down the road.
20:25:37 <smooge> weird
20:25:41 <skvidal> so the stages I liked best were
20:25:50 <skvidal> 1. move mailing lists over to collab
20:25:54 <nirik> #topic Planning/Brainstorming
20:26:46 <nirik> do we know how much space/resources that is taking up? less so without autoqa-results list...
20:26:48 <skvidal> 2. start prototyping new hosted instances using: $projectname.fedorahosted.org and a dns alias to point to the right places
20:28:01 <nirik> ok.
20:28:36 <nirik> so, we need to be migrated off the old machines by 2011-12-11.
20:28:41 <skvidal> wow
20:28:47 <skvidal> so we have what? 3 weeks?
20:28:47 <nirik> #info we need to be migrated off the old machines by 2011-12-11.
20:29:00 <CodeBlock> yow
20:29:00 <nirik> yeah, we might be able to ask for more.
20:29:11 <nirik> but they wanted to give us 1 month to move things.
20:29:50 <skvidal> ok
20:29:59 <nirik> so, we could do a phase 1.5: migrate everything to a new rhel6 instance setup basically like the current one.
20:30:07 <nirik> (or everything minus mailing lists)
20:30:11 <skvidal> okay
20:30:19 <skvidal> so 2 issues that plague me there
20:30:32 <skvidal> 1. we have no idea how much we trust/or not the new sb machines
20:30:40 <skvidal> 2. that whole network pain there
20:30:53 <j_dulaney> Why must we move?
20:31:03 <skvidal> j_dulaney: new machines
20:31:06 <nirik> j_dulaney: we have old hardware thats end of lifing...
20:31:12 <nirik> also, we really want to move from rhel5 to rhel6.
20:31:23 <nirik> which gives us new trac and lots of other things.
20:31:36 <skvidal> nirik: so - why don't we move some of them over to ibiblio02?
20:32:20 <skvidal> since we have that hw running
20:32:20 <skvidal> w/o any special network configurations
20:32:20 <skvidal> unless you're more confident in the nat thing
20:32:20 <nokia3510> is there going to be a testing of the machines prior to the actual migration ?
20:32:20 <nirik> thats an option, sure.
20:32:27 <nirik> Well, the nat thing seems to be working fine from all I can tell so far.
20:32:34 <nirik> rsyncs and backups have worked fine.
20:32:42 <nirik> but of course thats different than production.
20:34:14 <nirik> we could also do a 1.1 type action: move to a rhel5 on new machines (hosted02) for now, and have more time to migrate to rhel6 somewhere.
20:34:29 <nirik> but I guess it makes sense to do as much of the pain as we can stand now to get it over with.
20:34:46 <skvidal> nod
20:34:52 <abadger1999> eh
20:35:10 <nirik> I have collab03/04 I am going to build today (ready to commit to puppet and kickstart).
20:35:11 <abadger1999> if we have a deadline and we want reliability, 1:1 then followup with changes sounds better.
20:35:22 <abadger1999> otoh, if we don't mind solving bugs after migration
20:35:32 <abadger1999> front loading the pain is fine.
20:35:43 <abadger1999> what do our users expect from us?
20:36:04 <nirik> abadger1999: most probibly want stable hosting, but a number also want new trac/mailman/etc.
20:36:12 <nirik> so it's kinda split I think
20:36:25 * j_dulaney is +1 for pain now
20:36:29 <skvidal> abadger1999: the other issues is if we do rhel5
20:36:36 <skvidal> we'd be running rhel5 on rhel6 qemu
20:36:42 <skvidal> which, we know, is a recipe for..... no fun
20:36:53 <abadger1999> <nod> that's true
20:37:01 <nirik> skvidal: no, that way works fine I think... it's rhel6 on rhel5 xen that I thought was doom?
20:37:36 <abadger1999> so.. rhel5 on rhel6... is that just untested by us?
20:37:38 <nirik> we have several rhel5 app servers running on rhel6 kvm ok currently.
20:37:58 <abadger1999> k
20:38:09 <skvidal> ok
20:38:11 <nirik> app02/03/04/07 are all on kvm
20:38:48 <nirik> we could use the collab move as a way to determine if the machines are stable, but collab is pretty high profile, so I wouldn't want to break it much if we can avoid it.
20:39:43 <nirik> so, how would the $projectname.fedorahosted.org work?
20:39:48 <skvidal> cname
20:39:53 <j_dulaney> How long would it take to actually setup new RHEL 6 machines?
20:40:01 <skvidal> j_dulaney: they are up
20:40:04 <skvidal> the virthosts
20:40:17 <skvidal> the instances take like 20minutes or so - if you don't count all the random typing into puppet
20:40:28 <StylusEater> nirik: what about a low level box that is part of a load balancing cluster ... a web server?
20:40:36 <skvidal> nirik: cname: func.fedorahosted.org -> hostedsomebox01.fedoraproject.org
20:40:39 <nirik> ok, so, say we have project foo on hosted01 now. We move foo-devel list to collab, what else do we need to do to move foo to another machine/instance?
20:41:01 <skvidal> the git entries
20:41:04 <skvidal> people will have to switc
20:41:05 <skvidal> h
20:41:10 <skvidal> from git.fedorahosted.org
20:41:15 <skvidal> to $projectname.fedorahosted.org
20:41:22 <skvidal> that way we can move them wherever we want
20:41:24 <nirik> StylusEater: we talked about doing something like that at one point... not sure how much that gets us vs complexity.
20:41:35 <skvidal> and as along as the paths are consistent on the system it is pointed to
20:41:37 <skvidal> it shouldn't matter
20:41:43 <StylusEater> nirik: k.
20:42:09 <nirik> so everyone will need to change their repo config on their end user machine? any way we could avoid that?
20:42:12 <skvidal> the other advantage of simple dns aliases is this: we never have to play silly buggers trying to find the project machine
20:42:19 <skvidal> nirik: not that I'm aware of
20:42:46 <skvidal> right now EVERYTHING is $scm.fedorahosted.org
20:42:50 <skvidal> or fedorahosted.org
20:43:05 <nirik> yeah, and with http we could do a re-write, but with git/ssh, not so much.
20:43:07 <skvidal> I don't think we were planning on a lot of projects when it was originally designed
20:43:14 <skvidal> I don't know, though.
20:43:27 <skvidal> right - I'd expect us to do http rewrites
20:43:34 <skvidal> so people going to fedorahosted.org/$project
20:43:40 <skvidal> will get to the right host
20:44:09 <nirik> sure, but that doesn't help commits.
20:44:12 <j_dulaney> Maybe just http redirect pages?
20:44:21 <skvidal> nirik: I know
20:44:22 <StylusEater> skvidal: https://github.com/blog/530-how-we-made-github-fast
20:44:42 <nirik> could we have git spew out "HEY, please do git config blah, we have moved" ?
20:45:12 <skvidal> StylusEater: so enable host-authentication between systems for git over ssh?
20:45:24 <skvidal> StylusEater: I think I'd rather eat the flag-day change
20:45:57 <StylusEater> skvidal: no. I just read that the other day. Just another architecture that's in production.
20:46:22 <skvidal> StylusEater: this is the part we'd need to do
20:46:23 <skvidal> First, your Git client initiates an SSH session. The connection comes down off the internet and hits our load balancer.
20:46:23 <skvidal> From there, the connection is sent to one of the frontends where SSHD accepts it. We have patched our SSH daemon to perform public key lookups from our MySQL database. Your key identifies your GitHub user and this information is sent along with the original command and arguments to our proprietary script called Gerve (Git sERVE). Think of Gerve as a super smart version of git-shell.
20:46:55 <skvidal> Once access has been verified, Gerve uses Chimney to look up the route for the owner of the repository. The goal now is to execute your original command on the proper file server and hook your local machine up to that process. What better way to do this than with another remote SSH execution!
20:47:48 <nirik> so, say we do a rhel6/hosted03. We migrate projects to it on opt-in for a while and see if we run into stability issues or the like. Once we have a few projects working there, we should also be able to rsync/copy that to a hosted04 at ibiblio. In the event of problems we could just change dns and everyone goes there (provided it's synced)
20:47:57 <nirik> of course then we get the rsync time/issues between sites.
20:48:27 <skvidal> things to note - copy the ssh keys over
20:49:11 <skvidal> :)
20:49:13 * nirik ponders. The rsync backups locally to sb have been taking an hour or more, so I don't know that it's practical to keep two sites in sync...
20:49:36 <skvidal> if only there was some technology to keep 2 remote disks in sync
20:50:41 <nirik> well, there's a number, but I don't think any of them are magic. ;)
20:51:04 <StylusEater> nirik: or cost-effective
20:51:29 <nirik> well, the big problem is you only have so much BW... if you are making changes more than that... doom. But I'm not sure if we are or not.
20:51:39 <skvidal> ok
20:51:56 <skvidal> so - by switching to hostname-oriented projects
20:52:09 <skvidal> it means we can setup special use-case hosts and projects
20:52:17 <StylusEater> nirik: and there is going to be some latency unless the writes aren't confirmed until both sites are in sync
20:52:18 <skvidal> with different hosted software
20:52:21 <nirik> yeah, anywhere we like, which is great.
20:52:23 <skvidal> w/o having to cram it all on the same system
20:52:40 <skvidal> so - things like reviewboard
20:52:45 <skvidal> don't cause angst during updates
20:53:11 <skvidal> it means we no longer have to have a VERY SPECIAL script that handles git/bzr/hg/svn interfacing in ssh
20:54:26 <nirik> why is that?
20:54:35 <skvidal> b/c we don't HAVE to host bzr instances
20:54:37 <skvidal> git instances
20:54:39 <skvidal> hg instance
20:54:41 <skvidal> and svn instances
20:54:44 <skvidal> all o nthe same box
20:54:47 <skvidal> we may CHOOSE to
20:54:53 <skvidal> but we aren't stuck
20:55:20 <skvidal> let's say we're talking about one of abadger1999's projects
20:55:25 <skvidal> then we know it's using bzr
20:55:26 <nirik> sure, but we need some kind of script to limit them from doing more than commiting?
20:55:51 <skvidal> nirik: yes but it doesn't have to sift through all possible scms
20:55:55 <skvidal> that's all I mean
20:56:05 <nirik> ah, sure, true.
20:56:30 <abadger1999> eh -- six in one, half dozen the other.
20:56:59 <abadger1999> you still need to take multiple commands and make sure they're kosher... so you still have all the same boilerplate.
20:57:10 <nirik> ok, so I think we are all ok with moving mailing lists to collab? and I think we are all ok with moving projects to use a cname type setup so we can move them around when we want, right?
20:57:20 <skvidal> abadger1999: it seems likely we could reuse scripts from other orgsa
20:57:22 <abadger1999> actually... you probably want it all in one script
20:57:30 <skvidal> that are targetted at a specific scm
20:57:38 <skvidal> b/c they wisely chose a single scm
20:57:48 <skvidal> instead of embracing all of them
20:58:01 <abadger1999> skvidal: I don't know about that
20:58:12 * nirik thinks the scm thing is getting off into the weeds. ;)
20:58:13 <abadger1999> Until recently we were using separate scripts
20:58:21 <abadger1999> The main script was passing off o the other scripts
20:58:31 <abadger1999> subprocess()
20:58:40 <skvidal> nirik: yes
20:58:41 <skvidal> it is
20:58:46 <abadger1999> But we didn't get those from other sites
20:58:48 <skvidal> I'll withdraw the point
20:58:51 <skvidal> that's fine
20:58:56 <skvidal> I don't care that much right now
20:59:01 <abadger1999> (Which is not to say we might just have not looked0
20:59:04 <skvidal> can we agree on step1?
20:59:16 <skvidal> move collab and move the hosted lists over to there?
20:59:21 <abadger1999> +1
20:59:25 <nirik> yep. I think thats good.
20:59:40 <nirik> is everyone ok with me making collab03/04 at sb and pretty much as they are now, but rhel6?
20:59:45 <skvidal> +1
20:59:54 <abadger1999> +1
21:00:08 <nirik> I think (yes, it's me temping fate again) that it should be a pretty easy migration.
21:00:20 <nirik> the mailmail versions are not very different.
21:00:25 <nirik> mailman. sheesh
21:00:35 <skvidal> yah
21:00:46 <skvidal> we'll have to setup the multiple domains schtick for mailman
21:00:49 <skvidal> but that's not complicated
21:01:01 * skvidal has done it before
21:01:03 <skvidal> ssl can suck
21:01:05 <nirik> I think it's already that way in puppet... it has the domains listed at least.
21:01:06 <skvidal> but it's not that bad
21:01:09 <skvidal> is it?
21:01:12 <skvidal> if so - good
21:01:19 <nirik> yeah, not sure, but it seemed like it might be
21:01:40 <skvidal> I have to go AFK for a bit to do something before 5 happens and the whole world vanishes
21:01:44 <nirik> ok, next, is everyone ok with taking the pain with this migration to cname projects so we can later move them around?
21:01:44 <skvidal> so I'm going to do that right now
21:01:51 <nirik> ok. no worries.
21:02:23 * nirik is writing up some options to spew to the channel in a sec.
21:03:02 * nirik notes we have probibly bored everyone else by now. ;) Do ask questions if we aren't explaining what we are talking about...
21:03:13 <nokia3510> +1 from me, but the change should be announced rather soonish, imho
21:03:33 <skvidal> it would not be mandatory right away
21:03:37 <skvidal> we can EASE people into it
21:03:41 <nirik> yeah, agreed. we need to come up with a flag day schedule.
21:03:42 <skvidal> and then announce a flag day
21:03:48 * skvidal is really AFK now
21:06:44 <nirik> ok, so some options to spew out then... it seems to me we need to decide: rhel6 or a rhel5 until we migrate off it, in serverbeach or not, how many instances and if to do rsync/hotspare.
21:06:56 <nirik> 0. lists move to collab
21:06:56 <nirik> 1. projects get cnames in dns.
21:06:56 <nirik> 2. hosted03/04 at sb, 03 primary, 04 rsync copy.
21:06:56 <nirik> 3. hosted03/04 at sb, 03 and 04 both primary for some projects.
21:06:56 <nirik> 4. hosted03 at sb, hosted04 at ibiblio. One primary, other rsync spare.
21:06:57 <nirik> 5. hosted03 and hosted04 and hosted05 at sb, all primary for some projects.
21:06:59 <nirik> 6. Hosted02 at sb, rhel5, migrate from there.
21:07:01 <nirik> 6. Your plan here.
21:07:05 <nirik> oops... two 6's. ;)
21:08:32 <nokia3510> a guestimate of migrating from EL5 to EL6 would come in handy right now
21:08:52 <nirik> nokia3510: indeed. I think many things would just work...
21:09:04 <nirik> trac I think might take some db migration step, but I am not sure.
21:09:09 <nirik> git version should be around the same.
21:09:11 <abadger1999> How compatible is the new trac?
21:09:24 <abadger1999> That seems to be the major question about rhel5=>rhel6
21:10:15 * nirik digs around.
21:10:54 <nirik> http://trac.edgewall.org/wiki/0.12/TracUpgrade
21:11:47 <nokia3510> perhaps some el5 virtualized instances could run on the EL6 specific services as a temporary solution until the remaining issues are ironed out ?
21:12:21 <nirik> nokia3510: we could. It would likely then require another flag day down the road to get people off the rhel5 instance then.
21:13:09 <nokia3510> may be it would be the most time/cost effective way then...me thinks
21:13:12 <nirik> I'm sure there would be projects we never hear back from if we tried to get everyone to migrate.
21:13:57 <nirik> yeah, we have a hosted02 that I just installed a few days ago that is a rhel5 on the new hardware.
21:14:13 <nirik> so, if we went that route, we could use that machine to be the rhel5 catch all.
21:15:23 <abadger1999> So... from that doc it looks like upgrading should just work, right?
21:15:36 <nirik> abadger1999: yeah, famous last words, but yeah.
21:15:44 <abadger1999> in which case, I think we should do it.
21:16:00 <abadger1999> Maybe setup a rhel6 hosted box. Try to perform a mass upgrade.
21:16:06 <abadger1999> test the results.
21:16:15 <abadger1999> Fix what's broken.
21:16:17 * nirik notes that jlk had setup a test instance a while back, but can't recall what problems he saw on testing.
21:16:26 <StylusEater> abadger1999: makes sense
21:16:40 <abadger1999> Have a cutoff date to say -- unable to migrate now; migrate to new hardware with rhel5
21:16:48 <abadger1999> continue to test and fix.
21:16:55 <nirik> abadger1999: very reasonable.
21:17:12 <abadger1999> Then do it.
21:17:33 <abadger1999> That way we don't have to support a mixed rhel5/rhel6 hosted environment indefinitely.
21:17:55 <nirik> yeah, I personally would really like to move to 6, but I guess we should see how much pain is there right now.
21:18:08 <abadger1999> <nod>
21:18:42 <abadger1999> set it up and see I guess.
21:18:51 * nirik nods. types up timeline.
21:18:58 <nirik> I could do that in the next few days probibly.
21:19:43 <nirik> #info setup rhel6 box to test migrations with in the next few days.
21:19:59 <nirik> #info evaluate early next week if it's going to be smooth enough or not.
21:20:32 <nirik> #info setup collab03/04 later today
21:21:01 <nirik> When should we look at collab migration? possibly next week over the holiday? lists might be somewhat quiet then...
21:21:58 <nokia3510> quiet is good :)
21:22:03 <abadger1999> yeah
21:23:18 <nirik> as for flag day, we have a deadline on dec 11th, but thats a sunday. ;) So, perhaps go for the 7th?
21:23:32 <nirik> to give us a little slack in case.
21:24:59 <nokia3510> seems reasonable
21:27:34 * nirik gets a phone call
21:28:31 <abadger1999> Sounds good.
21:28:47 <nirik> perhaps friday would work for the collab migration...
21:28:57 * nirik isn't sure who will be around friday tho
21:29:07 <abadger1999> we also want a deadline of when we'll abort the rhel5=>6 migration and just work on old hw=> new hw migration.
21:29:39 <nirik> ha, it's a official rh holiday.
21:30:06 <nirik> I'd say mid next week... we should have a good idea by then if we are running into pain or not.
21:30:18 <nirik> 23rd?
21:30:50 <nirik> or perhaps the next week.
21:30:59 * nirik pulls up a calendar.
21:33:41 <nirik> I think the 23 might be enough. The thing is, if trac is easy we should know pretty quick.... and I don't think git or other things will be.
21:33:52 <nirik> if trac is a pain, we should know by then.
21:33:58 <abadger1999> sounds like a plan
21:34:33 <nirik> and for collab, perhaps the night of the 28th would be better than doing stuff on a holiday.
21:36:02 <nirik> doing stuff on a holiday might mean we change it over and think it's fine, and wander off, then it's really broken and we don't hear from users as fast/much.
21:36:11 <abadger1999> <nod>
21:36:36 <nirik> any other milestones we should have?
21:37:05 <nokia3510> the point of no return ?
21:37:16 <nokia3510> emailing the changes
21:37:28 <nirik> yeah, announcements. ;)
21:37:40 <nokia3510> that shoud be around 7th as well I presume
21:38:01 <nirik> I can announce the collab migration later today or in the next few days.
21:38:02 <nokia3510> should even
21:38:21 <nirik> we could also announce the hosted move for the 7th...
21:38:38 <nirik> but it might be good to have more info... ie, if we are going to rhel6 or not, etc.
21:39:03 <nirik> so, we could send that announce on the 23rd after we decide.
21:39:21 <nirik> oh, random thought I had a while back, not sure if it makes sense:
21:39:58 <nirik> find all admins of projects on fedorahosted. create list 'fedorahosted-announce', invite (not subscribe) all admins to it. use that list for hosted announcements instead of fedora-announce/devel-announce.
21:40:40 <nirik> might be too much trouble.
21:41:28 <nokia3510> maybe you could do the announce along with the migration stuff
21:41:45 <nokia3510> s/stuff/one/
21:42:02 <nokia3510> so they'd be notified already
21:42:04 <nirik> announce the announce list? ;)
21:42:37 <nokia3510> erm..yeah ? :)
21:43:08 <nokia3510> seems more like a split to me
21:43:36 <nirik> yeah, might just be not worth the effort. But some hosted people may not know they need to subscribe to fedora announce to see announcements...
21:44:27 <abadger1999> <nod> I think you're right about the latter.
21:45:20 <nirik> and also some fedora people may not really want to hear about hosted. ;)
21:45:33 <nirik> devel-announce seems like sorta a fit, but dunno
21:46:41 * nokia3510 thinks @fedorahosted and +fedorahosted would be also nice to have
21:47:25 <nirik> nokia3510: the announce list? or ?
21:47:51 <nokia3510> yes
21:48:29 <nirik> yeah, I waffle on if it will be useful, but I guess if it ends up getting no subscribers that would tell us. ;)
21:49:17 <nirik> we could announce the migration and the new list, and say future hosted announcements will be on fedorahosted-announce@xxxxxxxxxxxxxxxxxxxxxx, please subscribe if you care.
21:49:33 <nirik> and also tell new projects when we create them to subscribe/invite them to the list.
21:50:20 <nokia3510> nod
21:50:51 <nirik> ok, any other deadlines?
21:51:05 * nirik moves on to a project moving sop:
21:51:12 <nirik> - copy data to new machine. scm/trac/releases
21:51:12 <nirik> - create dns cnames for project: foo.fedorahosted.org pointing to the machine they are going to.
21:51:12 <nirik> - configure apache, migrate trac
21:51:12 <nirik> - Notify user they need to adjust scm setting to foo.fedorahosted.org/SCM/
21:51:28 <nirik> any other steps folks can think of?
21:52:46 <abadger1999> trac, web-scm browsers
21:52:48 * nokia3510 thinks that's about it for now
21:52:52 <abadger1999> all in the same sep
21:52:55 <abadger1999> *step
21:53:45 <abadger1999> nirik: Until we move to actually having separate machines, foo.fedorahosted.org/SCM/ and git.fedorahosted.org/git/SCM should both work right?
21:54:19 <abadger1999> (and also, some amount of time until we deprecate the old git.fh.o names)
21:55:02 <nirik> well, yeah... but... do we want a different flag day on that?
21:55:19 <nirik> because the ones that don't change, we can't easily move by themselves.
21:55:45 * nirik will be right back.
21:57:23 * nirik is back
21:59:24 <nirik> we could do it as a opt in type thing/new projects at first... but I think we do want to convert everyone so we can more easily move them.
21:59:28 <abadger1999> yeah -- I guess it makes sense to remove the SCM.fedorahosted.org stuff now.
21:59:29 <StylusEater> nirik: are there other items we're intentionally leaving off b/c they're puppet managed?
21:59:33 <abadger1999> all in one blow
21:59:43 <StylusEater> nirik: firewall configuration, keys, etc.
22:00:04 <nirik> StylusEater: yeah. That stuff should be taken care of by puppet. Although we may need to adjust things...
22:00:25 <StylusEater> nirik: k. just thinking through it as a newb to infra. :-)
22:00:31 <nirik> ie, right now we include all the scm groups on hosted, but if we moved off say 10 projects to their own instance, we might want to just include only those groups
22:02:31 <StylusEater> nirik: k.
22:02:39 * StylusEater is leaving work will be afk for a bit
22:04:36 <nirik> so, here's the dates I have:
22:04:58 <nirik> #info 2011-11-16 - finish creating collab03/04 instances.
22:04:58 <nirik> #info 2011-11-17 - rhel6 test instance for hosted to test with (hosted03?)
22:04:58 <nirik> #info 2011-11-23 - decide if we can migrate to rhel6 with hosted or not.
22:04:58 <nirik> #info (announce migration and flag day for the 7th announce announce list.)
22:04:59 <nirik> #info 2011-11-28 - test migrate some hosted instances/opt in
22:05:01 <nirik> #info 2011-11-28 - migrate to collab03/04 (evening)
22:05:03 <nirik> #info 2011-12-07 - flag day to migrate hosted
22:05:05 <nirik> #info url's change from fedorahosted.org/git/whatever -> project.fedorahosted.org
22:05:07 <nirik> #info 2011-12-11 - have to be done with serverbeach01-05
22:06:53 <nirik> so, I think whats left is to decide what instances where and how to backup/sync things. But we may need to defer that until we know if we are going to 6 or not.
22:09:34 <nirik> so, end users will need to change their scm on flag day, or sooner if they opt in... but thats the only end user action?
22:11:17 <abadger1999> nirik: Do we really want the test optin?
22:11:27 <nirik> not sure.
22:11:46 <nirik> I guess if we are operating on a copy of data, we already have all that...
22:11:53 <abadger1999> I guess it gives us real world usage for some things
22:12:21 <abadger1999> but it means we don't have a box that we can destroy at will anymore
22:12:24 <nirik> we can only really do that once we know if we are going with 5 or 6 and where.
22:12:32 <nirik> yeah
22:13:46 <nirik> once we decide that, we should have a instance(s), but then we have to be carefull not to overwrite or mess up things that have already moved before flag day.
22:13:54 <abadger1999> yeah
22:13:57 <nirik> (or move the bulk of them to another machine entirely from the earlybirds)
22:14:16 <abadger1999> That's my thought -- for instance, on migration day -- we might want to rsync data then run a script to update the trac dbs.
22:14:34 <abadger1999> If we have some real data already on the new box we'd have to make sure the script doesn't touch those.
22:14:40 <abadger1999> (and the rycn too :-)
22:14:42 <nirik> yeah.
22:15:01 <nirik> but on the other hand, having earlybirds could tell us if there are problems our testing didn't hit...
22:15:15 <abadger1999> <nod>
22:15:19 <nirik> but we could just ask them to hit the rhel6-test.fedorahosted.org and try
22:16:05 * nirik notes we may want to make sure mail is disabled on the test box.
22:17:03 <abadger1999> If we have test projects, maybe we can just backup their data as the first step of the upgrade.
22:17:17 <abadger1999> Then copy those test projects back into place as the last step
22:17:46 <nirik> I guess I am ok with a test box with the data and tell people to test against it with the understanding that all that data will be wiped with the production when we migrate... or before if we decide to...
22:18:15 <j_dulaney> Sorry, y'all, I had a huge distraction come up
22:18:18 * j_dulaney is back
22:18:48 <nirik> j_dulaney: no worries.
22:18:57 <j_dulaney> Where are we?
22:19:08 <nirik> advantage of just having a test box with all projects: we don't need to manage who is opting in or out of early testing.
22:19:22 <nirik> j_dulaney: mostly done... at least making progress. ;) read back up
22:21:09 <j_dulaney> nirik: Then who would be writing to said test box with all projects?
22:21:29 <nirik> we would be syncing the data over to it to test how migration worked out.
22:21:38 <j_dulaney> Righteo
22:22:00 * j_dulaney thinks we would need more comprehensive testing than that
22:22:17 <j_dulaney> Would test boxen have web interfaces, etc.?
22:22:33 <j_dulaney> And can we do test commits?
22:23:01 <nirik> yes, ideally it would have all that. Only thing that might be good is to drop email... otherwise it could mail about commits that never happened in the real repos.
22:24:20 <j_dulaney> Indeed
22:24:52 <j_dulaney> Has all this been discussed?
22:25:05 <nirik> all this?
22:25:18 <j_dulaney> Testing
22:25:41 <nirik> we are currently discussing if we should try and get earlybird projects to migrate and test, or just have a test instance with all data and ask everyone to test things against that.
22:26:12 <nirik> another problem with earlybirds would be if we decided rhel6 had too many issues to just migrate to... if we migrated them to rhel6, they couldn't easily be migrated back to 5.
22:27:52 <j_dulaney> True
22:28:16 <nokia3510> nirik: another potential deadline : if agreed to migrate to el6 _but_ something comes up requiring more time
22:28:21 * j_dulaney is wanting to lean torwards the second option and get everyone to test against it
22:28:34 * nirik is leaning that same way.
22:28:48 <nirik> nokia3510: yeah, we could always have to adjust... just knowing what we know now.
22:28:49 <nokia3510> all in less than a month ?
22:28:58 <j_dulaney> It's also more likely to catch oddities
22:29:30 <nokia3510> I lean too, but the deadline will most likely slip in that case
22:30:04 <nirik> nokia3510: yeah, it's tight for sure... but we should see issues pretty quick if they are big with it.
22:30:37 <j_dulaney> Just get as much of Dev to hop on testing as possible
22:30:59 * j_dulaney just remembered that he needs to change his SSH key
22:31:10 <nokia3510> otoh, getting things done now on el6 would be more appropriate on the long term, else 5>6 will require yet more time ... rather soonish I presume ?
22:31:21 <nirik> with as eager as some people have been to get to rhel6, I think we will get a lot of testers. ;)
22:31:59 <nokia3510> hoping on the holidays ?
22:32:33 <j_dulaney> That would be an added plus
22:33:07 <nokia3510> migrating without anyone to test you mean ? :)
22:33:10 <nirik> we can always re-evaluate how confident we are in the amount of testing at decision time.
22:33:42 <nirik> often in the open source world, there's _more_ people around on holidays as they get time off their dayjob to work on open source projects. ;)
22:34:17 <smooge> heh. honey.. you know that trip you wanted to take while RH was taking vacation...
22:34:24 <nirik> :)
22:34:33 <j_dulaney> LOL
22:34:49 <nokia3510> smooge: my point exactly :D
22:34:56 * j_dulaney may be with his GF and completely AFK
22:35:21 <smooge> well worse comes to worse I will just work from the RV
22:35:41 <smooge> need to figure out getting my android to do cell uplink
22:36:22 <nirik> so, if we go to a test box we ask people to test against, we probibly want to announce that as soon as we have the instance up with the data, right?
22:37:49 <nokia3510> that would mean late next week at the earliest ?
22:38:01 <j_dulaney> nirik: Indeed
22:38:08 <nirik> or friday hopefully.
22:38:19 <nirik> ok, here's what I have: http://fpaste.org/TVTy/
22:39:09 <nirik> corrections/comments/additions welcome
22:41:48 * nirik goes to let dogs out, back in a sec.
22:43:12 * j_dulaney really thinks that rhel6 would be the way to go
22:46:14 <j_dulaney> Could we put Trac on it's own RHEL 6 instance if we decide to go with RHEL5 elsewhere?
22:46:15 <nirik> yeah, if it doesn't cause too much pain.
22:46:23 <j_dulaney> Or would that be too much trouble?
22:46:38 <nirik> well, depends on if the thing keeping us on 5 was trac or not I guess. ;)
22:46:59 <j_dulaney> True
22:49:22 <nirik> ok, unless anyone has anything else I didn't cover, I think we can call it a day. ;)
22:50:15 <j_dulaney> Yay
22:50:33 <nirik> thanks everyone who came... lots of good feedback and ideas. ;)
22:50:39 <nirik> I think we have a somewhat solid plan now.
22:52:02 <nirik> thanks all! will mail what I have to the list.
22:52:05 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
