Also avaliable at: Minutes: http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.2009-06-11-20.00.html Log: http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.2009-06-11-20.00.log.html 20:00 < mmcgrath> #startmeeting 20:00 < fedbot> Meeting started Thu Jun 11 20:00:09 2009 UTC. The chair is mmcgrath. 20:00 < fedbot> Information about MeetBot at http://wiki.debian.org/MeetBot , Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00 * ricky 20:00 < mmcgrath> #topic Infrastructure -- Who's here? 20:00 -!- ChitleshGoorah1 [n=chitlesh@xxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:00 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Who's here? 20:00 * ricky 20:00 * LinuxCode 20:00 * SmootherFrOgZ is 20:00 * lmacken 20:00 * mmcgrath is here 20:00 * nirik is lurking around. 20:01 < mmcgrath> Ok, well lets get started 20:01 < mmcgrath> #topic Infrastructure -- F11 Release 20:01 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- F11 Release 20:01 -!- sijis [n=sijis@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:01 < mmcgrath> .ticket 1395 20:01 < zodbot> mmcgrath: #1395 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1395 20:01 < mmcgrath> So what went wrong with this release? 20:02 < mmcgrath> All in all I think it went very well. 20:02 < nirik> very little I think. ;) 20:02 < mmcgrath> The early bit flip (thanks f13) provided a 75% mirror rate when the anouncement went out. 20:02 * herlo is lurking 20:02 < mmcgrath> thats much better then for F10. 20:02 < ricky> I didn't expect docs links to change, so we had to do some last-minute updates that made us a bit late 20:02 < ricky> We'll make sure to double check that better next time around. 20:02 -!- ChitleshGoorah2 [n=chitlesh@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:02 < mmcgrath> Also, as mdomsch has pointed out many times, the torrents were screaming. The fastest torrent dl's I've ever had during a release. 20:03 * sijis is here. 20:03 < mmcgrath> 15-20 minutes for 4 isos compared to 8 or 9 hours previously. 20:03 * ianweller 20:03 * abadger1999 here 20:03 < nirik> yeah, due to them opening early we got a lot more seeders at release time. 20:03 -!- ldimagg__ [n=ldimaggi@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:03 < mmcgrath> ricky: should we make that a docs task or an us task? 20:03 < ricky> That'll be a websites task, I'll send links to the test page to list in advance. 20:04 * johe_ here too, by the way 20:04 < mmcgrath> k 20:04 < mmcgrath> The other thing that went kind of wacky was the bit leaks we had the thursday or friday before hand. 20:04 < mmcgrath> I think f13 is working on those though. 20:05 < mmcgrath> Oh, one other thing. 20:05 < mmcgrath> Proxy3 was hurting. 20:05 < mmcgrath> And I think I know why but I need to do a bit more research. 20:05 < mmcgrath> long story short it was doing too much. 20:05 < mmcgrath> I think it's just because it's one of our slower boxes in terms of cpu and disk. 20:05 < ricky> Did the x86_64 vs. i386 difference also come into play? 20:06 < mmcgrath> ricky: not this time. But proxy1 and proxy2 are kind of odd because they're behind a balancer. 20:06 * mmcgrath needs to fix that. 20:06 < ricky> Ah 20:06 < mmcgrath> But once the release was out the door.... Things went great. 20:06 < mmcgrath> Anyone have anything else on this topic? 20:06 < rjune_wrk> why are they behind the balancer, but the slowest box has to face the cold cruel world all alone? 20:07 < mmcgrath> rjune_wrk: well, we've got 4 sites, only one of which has a balancer. 20:07 < mmcgrath> so we've got 5 proxy servers, 2 behind the balancer, the others are all at the various sites. 20:07 -!- che [n=che@redhat/che] has joined #fedora-meeting 20:07 < rjune_wrk> ah, ok. 20:07 < mmcgrath> Ok, anyone have anything else? 20:07 < mmcgrath> Solid 20:08 < mmcgrath> #topic Infrastructure -- Fedora Community 20:08 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Fedora Community 20:08 < lmacken> We deployed our first TurboGears2 application this week :) 20:08 < mmcgrath> So fedora community shipped. 20:08 < mmcgrath> lmacken: you want to talk a bit about all of it? 20:08 < lmacken> Sure 20:08 < SmootherFrOgZ> \o/ 20:08 < lmacken> Software speaking, not much went wrong. 20:08 * mmcgrath is talking about https://admin.fedoraproject.org/community/ 20:08 < lmacken> I hit some insane traceback during staging that was caused by pointing to the unresolvabl 20:08 < lmacken> e db1.stg, instead of just db1. Other than that, everything went very smooth. 20:09 < lmacken> It's using both of our new memecached servers, using postgres on db2 for the feed store, and currently load balancing to mod_wsgi on app1 and 2 20:09 < lmacken> and it's the snappiest I've ever seen the app as of yet 20:09 -!- sseiersen|away is now known as sseiersen 20:09 < lmacken> but anyway, I did another update today to fix a variety of bugs, and we're going to keep cranking on it 20:09 < mmcgrath> It also means we can, at our leasure, convert our apps to TG2. 20:09 < lmacken> I want to plan a post-mortem when spot gets back from vacation 20:10 < lmacken> so we can talk about what happened, and how we can do it better 20:10 < mmcgrath> lmacken: is the whole tg2 stack in Fedora yet? I need to install and start playing with it. 20:10 < lmacken> the whole TurboGears2/FedoraCommunity/Moksha stack is in our fedora-infrastructure repo, and a chunk of it is still under review 20:10 < ricky> Does our tg2 support mean close-to automatic support for pylons as well? 20:10 < lmacken> ricky: yes 20:10 < ricky> Awesome 20:11 < lmacken> pylons should work fine now 20:11 < mmcgrath> excellent. 20:11 < mmcgrath> Anyone have any questions for Luke on fedora community? 20:11 < lmacken> I will start writing up a TurboGears2 SOP 20:11 < abadger1999> I'd like to get together some questions to ask spot wrt licensing. 20:12 < ricky> I guess that can be discussed at the post-mortem as soon as spot is back - can that be on IRC? 20:12 < abadger1999> That way we can have less round trips to legal. 20:12 < abadger1999> IRC +1 20:12 < mmcgrath> <nod> I just want to be better educated about it before I copy and paste :) 20:12 < lmacken> yes, it will be public, probably on irc or asterisk 20:13 < rjune_wrk> IRC+1 20:13 -!- ks3 [n=ks3@xxxxxxxxxxxx] has joined #fedora-meeting 20:13 < mmcgrath> Anyone have anything else on this? 20:13 < abadger1999> asterisk -1; IRC is better for public stuff. 20:13 < lmacken> mmcgrath: when do you wanna spread to app3-6? 20:13 < mmcgrath> lmacken: good work, thanks for the time you've put in on it. 20:13 < lmacken> abadger1999: agreed 20:14 < mmcgrath> lmacken: today or tomorrow. I know you did some updates, were they straight rpm updates? 20:14 < lmacken> mmcgrath: thank you. Thanks for your help as well, couldn't have done it without you guys 20:14 < lmacken> mmcgrath: yep, straight rpm updates 20:14 < mmcgrath> coolz. 20:14 < mmcgrath> ok so next topic 20:15 < mmcgrath> #topic Infrastructure -- Memcached 20:15 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Memcached 20:15 < mmcgrath> we've been slipping down the slippery slope that is memcached. 20:15 < mmcgrath> a month ago we were using it for sessions on the wiki. 20:15 -!- ldimaggi_ [n=ldimaggi@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed out) 20:15 -!- phunster [n=lanny@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit "Ex-Chat" 20:15 < mmcgrath> starting today we've got 3 apps using it and dedicated memcached servers. 20:16 < mmcgrath> the fas memcached stuff is going to exist just on the fas servers for security reasons. 20:16 < mmcgrath> everything else is on memcached[1-2] 20:16 < mmcgrath> ricky: do you want to remove memcached from puppet on app[1-2] or do you want me to 20:16 < mmcgrath> ? 20:16 < ricky> I'll get that after the meeting 20:16 < mmcgrath> solid 20:16 < mmcgrath> Nothing major here, just mentioning it. Any questions? 20:16 -!- ChitleshGoorah [n=chitlesh@fedora/ChitleshGoorah] has quit Read error: 113 (No route to host) 20:16 < ricky> Speaking of security, I just want to let everybody know that load access into memcached = arbitrary command execution with python-memcached 20:17 < ricky> So be careful about that :-) 20:17 < mmcgrath> <nod> 20:17 -!- djf_jeff [n=jeff@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit "I quit" 20:17 < mmcgrath> k 20:18 < mmcgrath> #topic Infrastructure -- fas update 20:18 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- fas update 20:18 < mmcgrath> So we just got done (well, are still doing) the fas updates. 20:18 < mmcgrath> This was a pretty major change as changes go. 20:18 < mmcgrath> Especially in how our shell accounts and aliases are generated. 20:18 < mmcgrath> So keep your eyes out for oddities. 20:19 < mmcgrath> So far things have been going fine, we've been running in staging for a while now without issue. 20:19 -!- ChitleshGoorah1 [n=chitlesh@xxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Connection timed out 20:19 < mmcgrath> I'm going to be moving our fas-sync to run more often on some of our more public servers 20:19 < mmcgrath> fedorapeople, fedorahosted, cvs, etc. 20:19 < mmcgrath> Any questions there? 20:19 -!- rdieter [n=rdieter@fedora/rdieter] has quit Remote closed the connection 20:19 < ricky> Woohoo :-) My goal is <= 5 minutes sync time 20:20 < mmcgrath> :) 20:20 < mmcgrath> Ok 20:20 < mmcgrath> #topic Infrastructure -- Outage this morning 20:20 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Outage this morning 20:20 < mmcgrath> So we had some doom this morning for those of you hanging out in #sysadmin-noc 20:20 < mmcgrath> Not sure what happened yet but I opened a bug about it 20:20 < mmcgrath> https://bugzilla.redhat.com/505352 20:20 < mmcgrath> xen13 up and died. 20:20 < mmcgrath> and took db2 with it, twice. 20:20 < mmcgrath> I moved db2 to a different host, hasn't happened since. 20:21 < mmcgrath> at the moment, nothing new on this so I'll move on. 20:21 < mmcgrath> #topic Infrastructure -- Pkgdb update 20:21 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Pkgdb update 20:21 < mmcgrath> abadger1999: we're on for a pkgdb rollout on Monday? 20:21 < abadger1999> that's right. 20:21 < mmcgrath> and this is a wsgi conversion? 20:21 < mmcgrath> any other changes? 20:21 < abadger1999> It's a major update that gets us using wsgi for the packagedb, all the bugfixes that accrued during change freeze, 20:22 < mmcgrath> I've still got smolt to convert. 20:22 < abadger1999> moving to usernames instead of userids i nthe public API. 20:22 < mmcgrath> after that I *think* all of our tg stuff will be good to go. 20:22 < ricky> Afdter pkgdb and smolt, will all of our apps be on haproxy + wsgi? 20:22 < abadger1999> Switching package status from the orphan owner to the status field. 20:22 < abadger1999> Lots of stuff changing. 20:23 -!- moixs [n=moixs@xxxxxxxxxxx] has quit Remote closed the connection 20:23 < lmacken> I'll get a new bodhi ready to go at that time 20:23 < lmacken> I should probably get a new one into staging now so we can test bodhi#319 20:23 < mmcgrath> ricky: I believe so. 20:24 < abadger1999> lmacken: Excellent 20:24 < mmcgrath> lmacken: sounds good. 20:24 < mmcgrath> Anything else on this? 20:24 < abadger1999> Test out admin.stg.fedoraproject.org/pkgdb ! ;-) 20:24 < mmcgrath> hehehe 20:24 < mmcgrath> #topic Infrastructure -- releng1.stg 20:24 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- releng1.stg 20:24 < abadger1999> I'm sure there's more bugs since this has major changes. 20:24 < ricky> After the pkgdb update, let's go through staging 20:24 < mmcgrath> speaking of staging 20:24 < mmcgrath> abadger1999: yeah :) 20:25 < mmcgrath> releng1.stg is nearing rediness. 20:25 < ricky> And make sure to get all the commits we want out of it before restarting the branch. 20:25 < mmcgrath> lmacken: I need to find out from you and jwb what is missing from it. I think it's going to take a close eye to make sure we know what we're putting in staging. 20:25 < mmcgrath> cron jobs and things that might fail, that sort of thing. 20:25 < lmacken> mmcgrath: readonly mount of /mnt/koji? 20:25 < mmcgrath> lmacken: it'll have that. 20:26 < lmacken> ok, to test mashing and whatnot, we'll have to tweak some paths 20:26 < mmcgrath> lmacken: and I don't think anything mash like is installed yet. 20:26 < mmcgrath> but I figure we'll work through it one boy one until it's usable for you and jwb. 20:27 < lmacken> ok, cool 20:27 < mmcgrath> I also assume you're going to need some sort of $TEMP_STORAGE 20:27 < lmacken> yeah 20:27 < mmcgrath> that's fairly large. we'll figure that out when the time comes. 20:27 -!- knurd is now known as knurd_afk 20:27 < mmcgrath> was jwb at FAD? 20:27 < lmacken> yep 20:27 < mmcgrath> probably traveling / busy 20:27 < mmcgrath> we'll get ahold of him. 20:27 < mmcgrath> ok, ricky :) 20:27 < mmcgrath> #topic Infrastructure -- Staging 20:27 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Staging 20:28 < mmcgrath> So we need to refresh the staging branch at some point soon. 20:28 < mmcgrath> and make sure all of our staging commits get merged back with master. 20:28 < mmcgrath> ricky: what are a rundown of some of the commits left? 20:28 -!- cassmodiah [n=cass@fedora/cassmodiah] has quit "Verlassend" 20:28 < mmcgrath> I know pkgdb had a bunch. 20:29 < ricky> Here's the full diff: http://ricky.fedorapeople.org/staging.diff 20:29 -!- phunster [n=lanny@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:29 < ricky> It's mostly pkgdb, I thnk 20:29 < ricky> **think 20:29 < ricky> It's not that bad, so we might get away with not blowing it away 20:30 < ricky> But I guess that's something to look at once pkgdb changes have been merged. 20:30 < abadger1999> pkgdb will be synced after Monday, so that's good. 20:30 < mmcgrath> Yeah 20:30 < ricky> Cool. 20:30 < abadger1999> Do we have a good recipe for merging the changes yet? 20:30 < ricky> And thanks to mmcgrath for starting to merge often this week 20:30 < mmcgrath> All in all, considering we don't really monitor this interaction much... things are in better shape then they otherwise could be. 20:30 < mmcgrath> ricky: figured i'd make a habit of it 20:31 < ricky> abadger1999: It'll probably consist mostly of finding out what we want to keep or throw away 20:32 < ricky> The process of merging isn't that painful at all, just getting updates on half-finished or abandoned work in staging. 20:32 < abadger1999> ricky: I'm just wondering, can I copy changes from staging to production, commit, and git will figure it out? Or do I need to use some git command to pick out specific changesets? 20:33 -!- Sonar_Guy [n=Who@fedora/sonarguy] has joined #fedora-meeting 20:33 < ricky> It's cleaner to cherry-pick, but I *think* hand-copying will work as well. I haven't tried doing work that way though, so I'm not sure how well it gets handled. 20:33 < ricky> I think the diff is short enough for us to go through it manually though. 20:33 < mmcgrath> I've hand-copied with success in the past. 20:33 < mmcgrath> but try to use git just so I'm mo'betta with it. 20:35 < abadger1999> Okay. We'll see what it looks like after Monday's merge ;-) 20:35 < mmcgrath> Ok, anything else on that? 20:35 < mmcgrath> yeah 20:35 -!- Sonar_Guy [n=Who@fedora/sonarguy] has quit Client Quit 20:35 < mmcgrath> With that, we'll open the floor 20:35 -!- Gremlin_ [n=lkrzemin@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:35 < mmcgrath> #topic Infrastructure -- Open Floor 20:35 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Open Floor 20:35 < mmcgrath> Anyone have anything they'd like to discuss? 20:35 < LinuxCode> apart from a million questions I have ? 20:35 < LinuxCode> lol 20:35 < ricky> Ask away :-) 20:36 < LinuxCode> I will have to figure out what is done for what purpose the enxt weeks 20:36 -!- Gremlin_ [n=lkrzemin@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Client Quit 20:36 -!- Sonar_Guy [n=Who@fedora/sonarguy] has joined #fedora-meeting 20:36 < LinuxCode> ricky, I ask you guys the next weeks, or we would be here all night 20:36 < LinuxCode> lol 20:36 < ricky> feel free to ask in #fedora-admin after the meeting as well 20:36 < LinuxCode> Id like to figure out how everything fits together 20:36 < LinuxCode> sure 20:36 < LinuxCode> I have one question though, the fas update was to kill bugs ? 20:37 < LinuxCode> it said so in the email I had 20:37 < abadger1999> LinuxCode: For some definition of bugs, yes. 20:37 < LinuxCode> abadger1999, I take it to make sure that people participate and not just sign up then ? 20:37 < abadger1999> Ther were bugs and there was also performance problems that were causing failures to occur. 20:37 < LinuxCode> ohh ok 20:37 < LinuxCode> i see 20:38 < ricky> The extra @fp.o aliases being created was a bug too 20:38 < LinuxCode> ricky, thats what I meant 20:38 < ricky> Unfortunately, it went unnoticed for a loong time. 20:38 < LinuxCode> would have been nice to give people a week though to sort themselves out 20:38 < ricky> The policy hasn't changed, it was just a bug that we weren't enforcing it. 20:38 < LinuxCode> I had the email today saying about the bug 20:39 < LinuxCode> hmm k 20:39 -!- sseiersen is now known as Video_juegos_not 20:39 < LinuxCode> well it doesnt matter now 20:39 < LinuxCode> for me anyway 20:39 < LinuxCode> ;-} 20:39 < mmcgrath> heheh 20:39 < mmcgrath> Anyone have anything else to discuss? If not we'll close the meeting early. 20:39 -!- Video_juegos_not is now known as sseiersen 20:40 < mmcgrath> Closing in 30 20:40 < mmcgrath> 10 20:40 < mmcgrath> #endmeeting
Attachment:
pgpYbzVqE40RU.pgp
Description: PGP signature
_______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
