============================================ #fedora-meeting: Infrastructure (2011-08-18) ============================================ Meeting started by nirik at 18:59:51 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2011-08-18/infrastructure.2011-08-18-18.59.log.html Meeting summary --------------- * Robot Roll Call (nirik, 18:59:52) * New folks introductions and apprentice tasks/feedback (nirik, 19:01:43) * F16 Alpha Freeze reminder and tickets. (nirik, 19:05:37) * Fedora hosted plans (nirik, 19:08:24) * Upcoming Tasks/Items (nirik) (nirik, 19:36:31) * RFR's (nirik, 19:43:52) * Meeting tagged tickets: (nirik, 19:47:30) * LINK: https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority (nirik, 19:47:30) * Open Floor (nirik, 19:49:29) Meeting ended at 19:56:52 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (130) * skvidal (116) * abadger1999 (21) * smooge (18) * rfelsburg (14) * zodbot (4) * Flare183 (2) * athmane (1) * CodeBlock (1) * pingou (1) * ricky (0) * codeblock (0) -- 18:59:51 <nirik> #startmeeting Infrastructure (2011-08-18) 18:59:51 <zodbot> Meeting started Thu Aug 18 18:59:51 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:59:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:59:52 <nirik> #meetingname infrastructure 18:59:52 <zodbot> The meeting name has been set to 'infrastructure' 18:59:52 <nirik> #topic Robot Roll Call 18:59:52 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999 18:59:52 <zodbot> Current chairs: abadger1999 codeblock nirik ricky skvidal smooge 18:59:58 * skvidal is here 19:00:00 * Flare183 is here 19:00:10 * athmane is here 19:00:16 <CodeBlock> hi, here, etc 19:01:32 <nirik> ok, I guess lets dive on in. 19:01:43 <nirik> #topic New folks introductions and apprentice tasks/feedback 19:01:58 <nirik> any new folks? or apprentice tasks we should discuss? 19:02:39 * nirik listens to crickets. 19:02:49 <skvidal> the ones here at night 19:02:51 <nirik> as always, feel free to chime in in #fedora-admin/noc. 19:02:52 <skvidal> they've been really loud 19:02:54 <smooge> How many items have been added to the talk repo 19:02:57 <rfelsburg> Not new persay, just back after being gone. 19:03:01 <smooge> the infra-repo 19:03:13 <nirik> skvidal: same here of late. 19:03:29 <nirik> smooge: infra-docs? or infra-hosts? 19:03:33 <smooge> geez I can't type.. infra-docs 19:03:36 <nirik> rfelsburg: welcome back. ;) 19:03:40 <smooge> welcome back 19:03:46 <nirik> smooge: some... lots more need to be done tho 19:03:49 <rfelsburg> thanks nirik/smooge 19:04:26 <nirik> I'd be happy to assist anyone wanting to work on that... 19:04:44 <smooge> me too. email works great if I am not on IRC 19:04:51 <skvidal> smooge: some - not all of them - I've been doing them as I read them 19:05:15 * nirik means to do some too, but hasn't. 19:05:34 <nirik> ok, moving along... 19:05:37 <nirik> #topic F16 Alpha Freeze reminder and tickets. 19:05:58 <skvidal> smooge: mainly b/c some of them I have to fix as I migrate 19:06:00 <nirik> I think sijis is working on getting the alpha website squared away. 19:06:06 <skvidal> smooge: b/c some of our SoPs are badly out of date 19:06:07 <skvidal> (sorry) 19:06:12 <smooge> np 19:06:17 * pingou hi 19:06:21 <nirik> yeah, many need updating. ;) 19:06:43 <nirik> we should have alpha syncing out soon, so we can check perms and space and all that fun stuff. 19:07:10 <nirik> anyone have any alpha related issues or concerns? 19:07:25 <smooge> no this one is pretty quiet on our side 19:07:31 <smooge> knock on silicon wood 19:08:04 <nirik> yeah, hopefully it will all go smoothly. 19:08:09 <nirik> ok, moving on then... 19:08:21 <nirik> oh, and hi pingou. ;) 19:08:24 <nirik> #topic Fedora hosted plans 19:08:26 <Flare183> I'll try to deal with the simple ones as needed 19:09:27 <nirik> we talked about this on the list some. I think we have at least a short term plan hashed out... make new collab/hosted pairs much like they are now, move lists to new collab, and migrate stuff to hosted with a eye for expansion down the road if need be. 19:09:32 <skvidal> Flare183: thank you 19:09:35 <abadger1999> smooge: I looked at allura's install doc... I think it would make hosted more heavy-weight than it currently is. Not sure if that's necessarily bad. 19:09:51 <nirik> Flare183: please do ask if anything looks unclear or out of date. ;) 19:10:08 <skvidal> Flare183: or migrate it and just mark it as OUT OF DATE at the top 19:10:21 * nirik has a dislike of sourceforge... possibly based on the old code, but it used to be really difficult to use. 19:10:21 <smooge> abadger1999, yeah.. this looks a lot more enterprise class than trac 19:10:29 <abadger1999> smooge: It might be hard to package... the download is a 25MB pybundle... whatever that is. 19:10:35 <skvidal> abadger1999: I thought the same thing when I looked at it - that allura looked like it had a lot of infrastructure we may or may not ever use ubt willl require a lot to keep it running 19:10:48 <skvidal> so 19:10:48 <abadger1999> <nod> 19:10:55 <skvidal> I actually have a ridiculous dream 19:11:01 <skvidal> if we could do redmine in openshift 19:11:06 <smooge> a git repo for every .pot? 19:11:12 <skvidal> and point it back at git repos on hosted 19:11:31 <skvidal> this is partly b/c I'm lazy and partly b/c I don't think it is a terrible thing to help openshift ;) 19:11:38 <abadger1999> <nod> 19:11:50 <nirik> yeah, worth exploring... 19:11:50 <skvidal> and b/c they are not bound by the 'need an rpm' rules 19:11:54 <skvidal> b/c of how they are laid out 19:12:06 <skvidal> anyway - I like the plan we've been discussing for hosted 19:12:12 <smooge> ok just as long we say this is an experiment and if openshift goes away for some reason we can roll with it 19:12:18 <nirik> does everyone agree on the short term plan? and we continue to look at longer term? 19:12:28 <skvidal> nirik: oh and even if we decide that merging hosted lists and collab lists is not a good plan - load-wise 19:12:39 <skvidal> nirik: I still think a single instance that just does lists for hosted saves us a bunch of pain 19:12:46 <skvidal> nirik: would you agree? 19:12:47 <nirik> yeah. 19:12:52 <nirik> agreed. 19:13:10 * nirik notes the archiver is still working on the 16th on hosted. Perhaps it will catch up someday. 19:13:21 <skvidal> nirik: right - do you know if it is cpu or io-bound?? 19:13:35 <abadger1999> nirik: What's the short term plan precisely? spinning new hosted boxes for backend and using dns to make it appear like one from the front? 19:13:52 <nirik> skvidal: cpu it looks like. 19:13:58 <abadger1999> is the splitting just for repos or for repos and trac? 19:14:05 <skvidal> nirik: can we throttle back other processes adn nice that one 19:14:28 <skvidal> abadger1999: i think it's for repos and trac? 19:14:40 <skvidal> note _think_ 19:15:22 <nirik> abadger1999: make new collab03/04, and hosted03/04 that are rhel6 and similar to what we have now. Migrate projects over, but add in cname or the like for trac/scm. It's still 1 machine, but we can easily move things around to 2 or more machines if need be down the road... 19:15:53 <nirik> and all lists migrate to the new collab03 19:16:23 <smooge> sounds good 19:16:26 <smooge> what else is on collab? 19:16:29 <abadger1999> okay. So actually. the initial change is very small. 19:16:30 <nirik> some of our load issues are mailman, some of them are our old trac/gitplugin. 19:16:34 <smooge> the older collab01/02 19:16:39 <nirik> so, new version might be much better. 19:16:39 <rfelsburg> I didn't think the archiver normally chewed that much on CPU nirik, but perhaps I'm just out of the loops, is this something new? 19:16:43 <abadger1999> the only thing splitting off is mailing lists. 19:16:49 <nirik> abadger1999: yep. 19:16:54 <nirik> and moving to rhel6. 19:16:57 <nirik> new trac, etc. 19:16:58 <abadger1999> The other portion is just a one-by-one migration to rhel6/new trac 19:17:11 <nirik> rfelsburg: well, we have this list... autoqa-results. 19:17:12 <abadger1999> Sounds good to me. 19:17:15 <nirik> it gets a LOT of emails. 19:17:17 <skvidal> nirik: question 19:17:28 <skvidal> do we want to go ahead and generate the cnames now? 19:17:31 <rfelsburg> Say no more nirik 19:17:37 <skvidal> point them all at hosted01 19:17:47 <skvidal> and then move them as we migrate hosts off? 19:17:51 <skvidal> s/hosts/projects/ 19:17:52 <skvidal> or 19:17:56 <skvidal> just add them as we migrate? 19:18:06 <nirik> skvidal: we could, but it might be better to add as migrated, so we have a good way to tell whats moved? 19:18:21 <nirik> rfelsburg: the 16th (which it's working on) had so far 11,000 emails. 19:18:25 <abadger1999> from a support pov, seems better to have the cnames now. 19:18:40 <abadger1999> so that supporting end users, we don't have to know what's moved. 19:18:42 <nirik> I guess it could be easy to tell from other ways. 19:18:44 <skvidal> abadger1999: b/c then we could tell everyone the same thing 19:18:50 <nirik> actually, we have the cnames now. 19:18:50 <abadger1999> exactly. 19:18:50 * skvidal is willing to do either :) 19:18:55 <skvidal> nirik: we do? 19:19:00 <rfelsburg> nirik: Geez, exceptionally bad day or has the email count really gone up that much. 19:19:07 <skvidal> nirik: I thought we just had a wildcard 19:19:15 <skvidal> rfelsburg: the latter 19:19:21 <rfelsburg> damn. 19:19:34 <nirik> skvidal: right, I was joking about the wildcard. They all resolve already. ;) 19:19:47 <nirik> due to wildcard 19:20:05 <skvidal> nirik: right - so we don't have the cnames now 19:20:07 <nirik> I guess I don't care strongly about making them now or later. 19:21:11 <nirik> I expect us to have a small pool of interested projects migrate soon... 19:21:22 <nirik> then move over time, then we set a flag day to move the laggers. 19:22:04 <rfelsburg> gotta love flag day 19:22:17 <nirik> Yep. ;) 19:23:15 <nirik> anyhow, anything more on the short term hosted plans? or long term? 19:23:39 <nirik> I'll note that I am also waiting to hear from serverbeach about a hardware refresh... would be nice to setup these new ones on new machines. 19:23:53 <skvidal> nirik: question 19:24:03 <skvidal> nirik: do we want to keep all of hosted in a single site colo-wise? 19:24:03 <nirik> or we could put them somewhere else. 19:24:15 <nirik> well, we could stick the secondary one somewhere else... 19:24:21 <nirik> it's a fair bit of BW probibly. 19:24:27 <nirik> but otherwise I don't see an issue... 19:24:28 <skvidal> ie: should we have some of them in ibiblio? 19:24:32 <skvidal> and some in osuosl, maybe? 19:24:50 <nirik> so, split them out now? 19:25:22 <skvidal> I was thinking maybe so? 19:25:34 <skvidal> do we gain anything by having them be on one site? 19:25:50 <nirik> well, ability to have the warm spare I suppose. 19:26:04 <nirik> or we could still do that on other sites, just more machines. 19:26:25 <nirik> I think osu is pretty low on disk 19:27:09 <skvidal> true 19:27:11 <nirik> I don't know how important the warm spare is. 19:27:19 <skvidal> one advantage of having lots of slices 19:27:23 <skvidal> for git repos/trac 19:27:29 <nirik> it means less downtime and less data lost in case of doom. 19:27:37 <skvidal> and less resource required to run a slice 19:27:52 <skvidal> once we have the base image 19:27:56 <skvidal> that we build from 19:28:12 <skvidal> I should be able to whip up something in func to select the slice with the least instances but most disk space 19:28:14 <skvidal> to allocate to 19:28:15 <abadger1999> we've also used the warm spare as a "staging" for hosted. 19:28:28 <abadger1999> which does kinda defeat using it as a warm spare, though :-( 19:28:28 <skvidal> abadger1999: to be fair we've done that kinda badly 19:28:47 <skvidal> abadger1999: the boxes are frequently not the same as one another 19:28:51 <abadger1999> <nod> 19:28:54 <skvidal> and if we want to test something out in the slice model 19:28:56 <nirik> well, we could still do warm spares/pairs... in more places. Just means more machines and such. 19:29:00 <skvidal> we can just have a slice 19:29:04 <rfelsburg> so turn it around, what are the benefits of moving it to osuosl or ibiblio 19:29:12 <skvidal> rfelsburg: I'm not suggesting moving it there 19:29:19 <skvidal> I'm suggesting allowing a slice to be there 19:29:23 <nirik> rfelsburg: diversity... if one site is down others are still up. 19:29:28 <nirik> less eggs in one basket. ;) 19:29:30 <skvidal> in other words do we need all of it in one colo 19:29:31 <abadger1999> we're juggling too many tasks with too few boxes... slices sound nice in that model. 19:29:37 <skvidal> abadger1999: +1 19:29:37 <rfelsburg> okay, that makes more sense. are we having any problems with reliability at this point? 19:29:48 <nirik> rfelsburg: nope, just paranoia. ;) 19:29:54 <skvidal> rfelsburg: all of our colos have had outages at one point or another 19:30:09 <skvidal> having it so all of any single resource is bound up behind a spof is, imo, a bad thing 19:30:13 <nirik> we do backup hosted to phx2, but thats daily... 19:30:25 <skvidal> we are trying to minimize the spofs 19:30:55 <rfelsburg> gotcha, sorry still trying to catch up to where we're at 19:31:27 <skvidal> rfelsburg: no sweat 19:31:29 <nirik> I guess the question is: do this now short term, or wait until we perhaps have a more cloudy setup later to better handle this... 19:31:44 <skvidal> nirik: with or without cloud instances 19:31:51 <skvidal> we need to setup the base slice 19:31:55 <nirik> right. 19:31:58 <skvidal> and figure out if this shit is gonna work at all :) 19:32:07 <nirik> so, how about this is 1.5? 19:32:18 <nirik> 1.0 -> rhel6 / base instance setup working. 19:32:33 <nirik> 1.5 -> duplicate on another site, and get working overall with the other one. 19:32:38 <nirik> 2.0 -> rainbows 19:33:02 <skvidal> I'm okay w/that 19:33:10 <skvidal> provided you add unicorns to 2.0 19:33:34 * skvidal is a holdout for unicorns 19:33:38 <nirik> once we have a hosted03 ready, it should be not too bad to duplicate it to a hosted05 at another site and migrate some things to it. 19:33:46 <nirik> aren't they messy tho? 19:33:48 <nirik> :) 19:33:51 <skvidal> nirik: they shit rainbows 19:33:53 <skvidal> so... no 19:34:04 <nirik> ha. 19:34:12 * skvidal is full of class today 19:34:13 <skvidal> okay 19:34:13 <nirik> ok, anything else on hosted? 19:34:23 <skvidal> do you care where hosted03 happens? 19:34:33 <nirik> I guess not... 19:34:33 <skvidal> or do we have a space for it at serverbeach? 19:34:44 <nirik> well, I wanted to wait and see if they were giving us new hardware. 19:34:55 <nirik> then we could setup the new stuff on the new hardware. 19:35:08 <skvidal> fair enough 19:35:13 * skvidal has no problem with waiting :0 19:35:14 <skvidal> :) 19:35:35 <nirik> I don't want to wait too long tho. RHEL6/trac0.12 has been waited for a long time. 19:35:57 <skvidal> nirik: agreed 19:36:29 <nirik> ok, moving along then... 19:36:31 <nirik> #topic Upcoming Tasks/Items (nirik) 19:36:41 <nirik> Freeze ends after the release next week. 19:36:52 <nirik> I have a few minor commits queued up. 19:37:12 <nirik> I'd like to get a rhel6 app server setup 19:37:24 <nirik> switch some backups to new backup box 19:37:48 <nirik> possibly look at re-installing tummy01 or one of the smaller remote hosts. 19:38:12 <nirik> I think we could look at migrating ibiblio01 stuff to new rhel6 versions on ibiblio02 19:38:34 <nirik> any folks have other items waiting for after the freeze? 19:38:48 <nirik> beta freeze starts 09-13 19:40:17 <nirik> we have some updates piled up... none of them seem reboot requiring. 19:40:45 <skvidal> always nice to hear 19:41:04 <nirik> although I am sure there will be a os minor release sometime to mess us up. ;) 19:41:48 <nirik> Of course I'd like to get the new boxes installed and usable hopefully soon too. ;) 19:42:00 <nirik> anyhow, if no one has any exciting plans, will move on in a min 19:42:41 <abadger1999> I've got the ambassadors raffle app about ready -- just suddenly realized that I was using CSS code from fedoracommunity and it's agpl so I have to fix that somehow before putting it someplace public for inode0 to evaluate. 19:42:52 <nirik> cool. 19:42:55 <skvidal> abadger1999: :( 19:42:58 * abadger1999 sent email to mizmo to see if she can relicense 19:43:09 <abadger1999> otherwise, I guess I throw out the css and reimplement clean. 19:43:26 <nirik> :( 19:43:36 <abadger1999> nirik: how's askbot coming? 19:43:45 <nirik> oh yeah, I was going to discuss that some. 19:43:52 <nirik> #topic RFR's 19:44:03 <nirik> so, askbot is looking to move to staging now... 19:44:15 <nirik> I posted to the list a idea/plan for setting it up. 19:44:31 <nirik> basically: add to proxy01.stg and make a new ask01.stg instance for backend. 19:44:49 <nirik> I am not sure if it makes sense to have it's db on the main db host or seperate on it's own instance. 19:45:07 <nirik> ask does need rhel6. 19:45:34 <nirik> so, feedback on that welcome. 19:46:10 <nirik> I think fpaste is still working on their dev instance to get everything aligned and ready... 19:47:17 <nirik> Hopefully people will like askbot and use it. :) 19:47:30 <nirik> #topic Meeting tagged tickets: 19:47:30 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority 19:47:37 <nirik> any other tickets folks would like to discuss? 19:48:35 <smooge> not me.. I am still giggling about unicorn farts 19:48:42 <skvidal> smooge: unicorn poop 19:48:44 <skvidal> get it right 19:48:52 <nirik> just as a side note, I'd like to get us under 200 tickets by the end of the year. ;) We will see... 19:48:55 <skvidal> unicorn farts are just methane like anyone else :) 19:48:57 <smooge> no I was thinking that if they poop rainbows what do they fart? 19:49:09 <nirik> skvidal: thats boring. ; 19:49:10 <skvidal> smooge: it's colorful methane, I admit 19:49:25 <skvidal> nirik: rainbow colored methane is boring? 19:49:27 <skvidal> hrmph 19:49:29 <nirik> #topic Open Floor 19:49:38 <nirik> Anyone have anything for open floor? 19:49:41 <smooge> I figure they farted lasers 19:49:48 <smooge> methane powered lasers 19:49:52 <nirik> perhaps ponys? 19:50:02 <skvidal> nirik: they fart ponies? 19:50:07 <nirik> yeah. ;) 19:50:08 <skvidal> dear god that would hurt 19:50:13 <nirik> I'd say so. 19:50:21 <skvidal> open floor 19:50:28 <skvidal> I emailed the epylog people 19:50:35 <skvidal> and they said "what? oh that? yah, okay" 19:50:52 <skvidal> so I'll probably rip it away from svn with all of its weirdnesses 19:50:58 <skvidal> and put it into git with all of its weirdnesses 19:51:03 <nirik> excellent. 19:51:04 <skvidal> but weirdnesses I have to deal w/more often 19:51:10 <nirik> would be good to upstream things. 19:51:11 <skvidal> and merge all the modules and patches I have 19:51:39 <nirik> any further thoughts on the auth issue? 19:51:47 <skvidal> that I hates it? 19:51:47 * nirik didn't come up with any brilliant solution. 19:52:44 <skvidal> if nothing else 19:52:48 <skvidal> I will make an htpasswd file 19:52:51 <skvidal> in the dir 19:52:58 <skvidal> and chown it to sysadmin-logs 19:53:04 <skvidal> and you can add your damned self 19:53:12 <nirik> sure. simple. I like it. 19:53:25 <skvidal> and to be fair 19:53:28 <skvidal> if you can't add yourself 19:53:33 <skvidal> then you don't need to look at the logs ; 19:53:34 <skvidal> ) 19:54:09 <nirik> sounds fine. might add it to the log sop too then. 19:54:16 <skvidal> indeed 19:54:18 <rfelsburg> sounded a little jaded there skvidal lol 19:54:24 <skvidal> rfelsburg: a little? 19:54:27 <skvidal> hmm 19:54:31 <rfelsburg> tin y bit :) 19:54:32 <skvidal> I must be softening in my old age 19:54:36 <skvidal> I meant to sound a lot jaded 19:54:36 <skvidal> :) 19:54:44 <rfelsburg> lol 19:54:53 <skvidal> just in case it isn't clear 19:54:56 <skvidal> </sarcasm> 19:54:58 <skvidal> I'm just messing about 19:55:01 * nirik will close out the meeting in a minute or two if nothing else comes up. 19:56:13 <rfelsburg> If I didn't understand sarcasm I'd have no place in IT 19:56:47 <nirik> ok, thanks for coming everyone! 19:56:52 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
