20:02 < smooge> #startmeeting Fedora Infrastructure 2010-03-11 20:02 < zodbot> Meeting started Thu Mar 11 20:02:43 2010 UTC. The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:02 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:02 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic: Fedora Infrastructure 2010-03-11) 20:02 < smooge> thankyou zodbot 20:03 < smooge> mmcgrath is finishing up a meeting so I am just helping along 20:03 < smooge> #topic Roll Call 20:03 -!- zodbot changed the topic of #fedora-meeting to: Roll Call (Meeting topic: Fedora Infrastructure 2010-03-11) 20:03 * ricky 20:03 < smooge> smooge 20:03 * a-k is here 20:03 * nirik is around. 20:04 < smooge> mdomsch said he was here a second ago before I started meeting 20:04 -!- rdieter_ [~rdieter@xxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:04 < smooge> mmcgrath will be here soon. 20:04 -!- biertie [~bert@fedora/biertie] has joined #fedora-meeting 20:04 -!- rdieter [~rdieter@fedora/rdieter] has quit Ping timeout: 265 seconds 20:04 * ayoung is here 20:04 < smooge> ok let me pull up the other standard things we do in the meeting 20:05 < smooge> #topic Meeting Tickets 20:05 -!- zodbot changed the topic of #fedora-meeting to: Meeting Tickets (Meeting topic: Fedora Infrastructure 2010-03-11) 20:06 < smooge> According to trac.. there are no tickets for this meeting :) 20:06 < smooge> am i missing anything people know? 20:07 < smooge> ok moving on 20:07 < smooge> #topic Alpha Release - https://fedorahosted.org/fedora-infrastructure/report/9 20:07 -!- zodbot changed the topic of #fedora-meeting to: Alpha Release - https://fedorahosted.org/fedora-infrastructure/report/9 (Meeting topic: Fedora Infrastructure 2010-03-11) 20:07 < smooge> alpha release was on Tuesday 20:08 < smooge> ok tickets 20:08 < smooge> sorry for my slowness guys.. my greps aren't as fast as I hoped 20:08 < smooge> .ticket 1944 20:08 < zodbot> smooge: #1944 (Fedora 13 Alpha Partial Infrastructure Freeze 16/Feb - 3/March) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1944 20:08 < a-k> # topic? 20:09 < mmcgrath> yo 20:09 * mmcgrath here :) 20:09 < smooge> hi mmcgrath 20:09 < mmcgrath> herro 20:09 < mmcgrath> smooge: you want to keep going or you want me to take over? 20:09 < smooge> hopefully I am helping here ... but its been real quiet. y 20:09 -!- gholms|work [~gholms@xxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:09 < smooge> you can take over... 20:09 < mmcgrath> alrighty 20:09 < smooge> you get people to say things :) 20:10 < smooge> #chair mmcgrath 20:10 < zodbot> Current chairs: mmcgrath smooge 20:10 < mmcgrath> hehehe 20:10 < mmcgrath> so this alpha release went fine, but with oddities. 20:10 < mmcgrath> I'll go ahead and close 20:10 < mmcgrath> .ticket 1944 20:10 < mmcgrath> and 20:10 < zodbot> mmcgrath: #1944 (Fedora 13 Alpha Partial Infrastructure Freeze 16/Feb - 3/March) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1944 20:10 < mmcgrath> .ticket 1990 20:10 < zodbot> mmcgrath: #1990 (Release Day Ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1990 20:10 < mmcgrath> lets talk about .ticket 1992 20:10 < mmcgrath> .ticket 1992 20:10 < zodbot> mmcgrath: #1992 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1992 20:11 < mmcgrath> So the first problem we actually ran into was that bapp1 had puppet disabled. 20:11 < mmcgrath> ricky: you around by chance? 20:11 < ricky> Yup 20:11 < mmcgrath> so having puppet disabled on bapp1 did what exactly? 20:12 < ricky> syncStatic is run by the apache user on bapp01 now, so that prevented it from getting the change to update the website 20:12 < mmcgrath> got'cha, so new syncStatic didn't make it on to the server so new website didn't make it to the proxy servers. 20:12 < mmcgrath> this is a monitoring thing, so when we get to monitoring which servers have puppet disabled, that one will go away 20:12 < mmcgrath> The other thign that happened was with our i2 netapp 20:13 < mmcgrath> Smooge and I just got off a meeting with the storage team about this. 20:13 < mmcgrath> basically it took several hours to transfer 16G worth of blocks. 20:13 < mmcgrath> the temporary fix was to put our sync traffic at a higher QoS then other traffic 20:13 < mmcgrath> long term though there was actually something wrong with the link... again. 20:13 < mmcgrath> they know and they're working on it 20:14 < mmcgrath> but it's certainly something we're going to want to track ourselves. 20:14 < mmcgrath> But, really the alpha went well even with those things... for non i-2 users anyway. 20:14 < mmcgrath> we had 85%+ good mirror rate 20:15 < mmcgrath> And the last thing I'd say is boot.fedoraproject.org 20:15 < smooge> and once the I-2 got up I think they had little issues too 20:15 < mmcgrath> Oxf13: ping 20:15 < mmcgrath> smooge: indeed 20:15 < mmcgrath> well, right now I just did boot.fedoraproject.org 20:16 < mmcgrath> but it dawns on me that's sort of a releng task. 20:16 < mmcgrath> one that I'm happy to do 20:16 < mmcgrath> but I wanted to check with Oxf13 as to where he things the SOP should sit, in Infrastructure or RELENG. 20:16 -!- kedars [~kedars@xxxxxxxxxxxxxxxxxxxx] has quit Ping timeout: 276 seconds 20:16 < mmcgrath> it's a minor distinction in this case, but making one group responsible will ensure that it always gets done :) 20:16 < mmcgrath> So anyway 20:17 < mmcgrath> anyone have any other questions about the alpha release? 20:17 < mmcgrath> alrighty 20:17 < mmcgrath> well 20:17 < mmcgrath> #topic Next freeze 20:17 -!- zodbot changed the topic of #fedora-meeting to: Next freeze (Meeting topic: Fedora Infrastructure 2010-03-11) 20:17 < mmcgrath> The next freeze is coming upretty quick 20:17 < mmcgrath> #link http://fedoraproject.org/wiki/Schedule 20:18 < mmcgrath> Infrastructure will be freezing on the 23rd. 20:18 < mmcgrath> that's less then 2 weeks. 20:18 < mmcgrath> so keep that in mind as you're deploying new things. 20:18 < smooge> so we should go slushy 20:18 < smooge> ? 20:18 < mmcgrath> slushy? 20:19 < ricky> Ice, slushy, freeze? :-) 20:19 < mdomsch> I have some MM fixes to get out, but may not make it before the next freeze :-( 20:19 -!- rdieter_ is now known as rdieter 20:19 < mmcgrath> mdomsch: anything I can do to help? 20:19 * mdomsch needs to test/fix one part, but it's been a month, so I forget which part it was ;-( 20:20 < smooge> sorry I meant, if you are going to be testing stuff for over 2 weeks.. not the time to do it. 20:20 -!- kedars [~kedars@xxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:20 < mmcgrath> yeah 20:20 < mdomsch> mmcgrath, nothing huge; I'll get to it, or not... 20:20 < mmcgrath> I'll try to send a couple of reminders as the time gets closer. 20:20 < mmcgrath> the freeze always sneak up on us :) 20:21 < mmcgrath> anyone have anything else on that? 20:21 < smooge> we have updates to do this/next week 20:21 < mmcgrath> skvidal: ping 20:21 < skvidal> pong 20:21 < mmcgrath> smooge: lets talk about that 20:21 < mmcgrath> #topic Monthly Update 20:21 < smooge> okie dokie 20:21 -!- zodbot changed the topic of #fedora-meeting to: Monthly Update (Meeting topic: Fedora Infrastructure 2010-03-11) 20:21 < mmcgrath> skvidal: ok, so the list I've been working from with you is very nearing completion. 20:21 -!- rwmjones [~rjones@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Ping timeout: 256 seconds 20:22 < mmcgrath> skvidal: think we'll be in any place to update soon? 20:22 < skvidal> yah - I think likely 20:22 < skvidal> though not sure TOMORROW is gonna happen 20:22 < smooge> thats fine tuesday sound ok? 20:23 < skvidal> sounds possible - depends what gets set on fire over the weekend 20:23 < mmcgrath> tuseday of next week sounds good, the freeze starts one week after that. 20:23 < smooge> yeah which was why I didn't want monday :) 20:23 < skvidal> unless there are A LOT of yum bugs that show up in f13a over the weekend 20:23 < skvidal> then I should have the time 20:23 < smooge> wghat are the changes you are working on? 20:24 < mmcgrath> smooge: I've been removing old hosts from puppet 20:24 < mmcgrath> skvidal's been working on having func use those hosts and coming up with a solid update script 20:24 < mmcgrath> or func program 20:24 < skvidal> smooge: you talking to me or mmcgrath? 20:24 < mmcgrath> I'm not sure what they're called. 20:24 < skvidal> okay 20:24 < skvidal> so here's all it ids 20:24 < skvidal> err is 20:25 < skvidal> 1. make it so our func certs aren't constantly screwed up 20:25 < smooge> ah 20:25 < skvidal> 2. make it so our func minions match our puppet minions 20:25 < skvidal> 3. have a script to let us do searches/installs/updates/etc from puppet1 w/o having to schlep all over the place 20:26 < skvidal> the script I worked on a couple of weeks ago 20:26 < skvidal> and got something mostly functional - but with room for improvement 20:26 < mmcgrath> smooge: do you know if this update will require a reboot? 20:26 < skvidal> the first 2 is what I spent the week working on to solve the problem that our func minions were completely wrong 20:26 < skvidal> and mangled badly from the phx2 move + rename stuff 20:27 < smooge> well it depends on if a kernel update gets dropped over the weekend :). At the moment I don't think so 20:27 < mmcgrath> yeah so I've been most of my last afternoon and this morning renaming hosts. I think I've got 3 hosts left and a little additional cleanup to do 20:28 -!- bpepple|lt [~bpepple|l@xxxxxxxxxxxxxx] has joined #fedora-meeting 20:28 < mmcgrath> OK 20:28 < smooge> no the worst will be an openssh restart 20:28 < mmcgrath> so anyone have anything else on this? 20:28 < mmcgrath> <nod> 20:28 < skvidal> umm 20:28 < skvidal> I have a couple more things 20:28 < mmcgrath> skvidal: have at it 20:29 < skvidal> so I'll see if I can get a new func pkg out for all the hosts and a mechanism to update their minion.conf files to point to the puppet certificates 20:29 < skvidal> that's going to be the REALLY fun part :) 20:29 < mmcgrath> If you need any puppeting help let me know 20:30 < mmcgrath> you talking about a new func epel package or in the infra repo/ 20:30 < skvidal> is minion.conf on the boxes puppet controlled now? 20:30 < skvidal> infra repo 20:30 < skvidal> for now 20:30 < skvidal> it'll eventually make it over to epel 20:30 < skvidal> but this is new cod 20:30 < ricky> I think so 20:30 < skvidal> e 20:30 < smooge> and if it isn't I will be happy to schlep what needs to be done 20:31 < skvidal> anyway - that's really all 20:31 < skvidal> the func-yum overlord script is fairly simple and could be added to by anyone as we go along 20:31 < skvidal> that's what's going to do the update/list updates/etc work 20:31 < gholms|work> I just wanted to mention that that is the greatest script name ever. 20:32 < mmcgrath> skvidal: and just becuase I'm overly paranoid... the time out bug? all fixed in the new version? 20:32 < skvidal> mmcgrath: the timeout bug was fixed long long ago 20:32 < skvidal> we've been running an old version of func for quite sometime 20:33 < dgilmore> sorry im late 20:33 < mmcgrath> k 20:33 < mmcgrath> dgilmore: no worries 20:33 < skvidal> dgilmore: BETRAYER! 20:33 < smooge> the only thing I wanted to ask was what would be prefered for making func 'groups/classes' 20:33 * skvidal giggles 20:33 < smooge> skvidal, hey hey hey.. thats me 20:33 < mmcgrath> Ok, anyone have anything else on this? If not we'll move on. 20:33 < skvidal> dgilmore: sorry, I just thought it was hilarious! 20:34 < dgilmore> skvidal: i had to get food for dinner and screwed up times 20:34 < smooge> nothing form me 20:34 < mmcgrath> ok 20:34 < mmcgrath> #topic Search Engine 20:34 -!- zodbot changed the topic of #fedora-meeting to: Search Engine (Meeting topic: Fedora Infrastructure 2010-03-11) 20:34 < mmcgrath> a-k: ping, want to take this? 20:34 < a-k> Sure 20:34 < a-k> I looking again at one of the candidates I eliminated from consideration earlier 20:34 < a-k> It advertises the most complete support for Unicode char sets 20:35 < a-k> I had eliminated it because I couldn't get it to work with SQLite 20:35 < a-k> ... and its user forum had no answered questions about how to get SQLite to work 20:35 < mmcgrath> which one was this? 20:35 < a-k> mnoGoSearch 20:35 < Oxf13> pong 20:35 < a-k> So I'm not there yet, but is there a db server I can use in pub test or do I need to install my own? 20:36 < a-k> Either MySQL and PostgreSQL should work 20:36 < mmcgrath> a-k: we'll have one for when we move to staging and production, but on the pt servers, just yum and install one 20:36 < a-k> OK. I'm still working on it locally, but I'll move to pt when I'm ready 20:36 < mmcgrath> cool 20:36 < a-k> I think that's it for now 20:37 < mmcgrath> a-k: thanks 20:37 < mmcgrath> Anyone have any questions for a-k about that? 20:37 < dgilmore> a-k: do we really care for sqlite support? 20:37 < a-k> I thought SQLite would be easier/preferable to MySQL or PostgreSQL 20:38 < mmcgrath> easier for a demo but probably not for production 20:38 < a-k> None of the other candidates had needed an external db 20:38 < mmcgrath> interesting 20:38 < mmcgrath> they all had their own local store then? 20:38 < a-k> The other ones use their own local db 20:38 < mmcgrath> yeah 20:38 < mmcgrath> alright, anyone have anything else? 20:39 < mmcgrath> alrighty 20:39 < mmcgrath> #topic Monitoring 20:39 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic: Fedora Infrastructure 2010-03-11) 20:39 < mmcgrath> SO we talked about this on infrastructure for a bit. 20:39 -!- sonargal [~Test@fedora/SonarGal] has quit Quit: used jmIrc 20:39 < mmcgrath> it leaves us in an ackward position 20:39 < mmcgrath> do we just dump zabbix now? 20:39 < smooge> mmcgrath, did you want to get Oxf13 while he was here. 20:39 < mmcgrath> go back to nagios for a bit? 20:39 < dgilmore> mmcgrath: yes 20:39 < mmcgrath> smooge: oh right 20:40 < mmcgrath> Oxf13: did you want me to put the boot.fedoraproject.org as a releng SOP or an infrastructure SOP? it seems more releng, I'm happy to do it for as long as I'm part of Fedora but it should be documented :) 20:40 < Oxf13> that's a good question. 20:40 < Oxf13> it does sound relengy 20:41 < mmcgrath> it's pretty easy to maintain, I just want to make sure it actually gets done every release. 20:41 < mmcgrath> Oxf13: you're call. I'll write it up this week sometime, just let me know :) 20:41 * dgilmore thinks is a releng thing 20:42 * mdomsch says releng ;-) 20:42 < mmcgrath> alrighty, well unless Oxf13 says otherwise I'll put it as a marketing sop :-P 20:42 < mmcgrath> naw, I'll just put it in releng for now and if we change our minds later we can move it. 20:42 < mmcgrath> ok, back to monitoring. 20:42 < mdomsch> mmcgrath, unless you write a script to run when releng pushes a tree, to update bko automatically. then it's an infra ticket that gets opened and closed automagically 20:42 < ricky> I think zabbix has been taking away effort on improving our nagios monitoring, so I'm up for dumping 20:43 < Oxf13> releng works for us, and if you come with content that's even better (: 20:43 < mmcgrath> ricky: so the problem then is trending. 20:43 < mmcgrath> our cacti install 20:43 < mmcgrath> doesn't even seem to exist anymore :) 20:44 < ricky> Apart from request times for our websites, what do you want to be able to monitor that cacti can't? 20:44 < smooge> can I work on tht with someone? 20:44 < ricky> Well, easily can't 20:44 < mmcgrath> ricky: well, the problem is that we have to enter data in two locations 20:44 < mmcgrath> and that has inherit problems. 20:44 < mmcgrath> and doing custom trending in cacti can be tricky at times. 20:45 < ricky> What kind of problems? You still get your notification when something goes down and the general idea of movements, right? 20:45 < mmcgrath> anyway, lets take a look at our current zabbix install 20:45 < mmcgrath> err nagios install 20:45 < mmcgrath> and see how it goes. 20:45 < mmcgrath> ricky: ehh, I use trending a lot, to see when things started. 20:45 < mmcgrath> the alert is when a threshold started, but it's usually only a small part of the story. 20:45 < ricky> I've also not been crazy about running the zabbix agent public facing either. 20:46 < mmcgrath> for example when MM had the bloated pickle issues. 20:46 < mmcgrath> we didn't get the alert until days after MM was upgraded. 20:46 < mmcgrath> without trending we wouldn't have noticed when the problems started 20:46 < ricky> Yeah, but cacti would still have given you the big picture you needed, right? 20:46 < dgilmore> ricky: i rember you disabled catci 20:46 < mmcgrath> if it were in there. 20:46 < mmcgrath> trying to keep cacti and nagios in sync is goign to be a pretty big pain. 20:47 < mmcgrath> especially when we start wanting to monitor artibrary bits of info with cacti 20:47 < smooge> ricky, that is my major problem with it also 20:47 < mmcgrath> it gets complex pretty quick 20:47 < smooge> I think we should have one agent per server.. and thats func 20:47 < ricky> dgilmore: Probably, I don't running public facing stuff we're not using :-) 20:47 -!- rwmjones [~rjones@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:47 < dgilmore> ricky: from memory there was a security bug in cacti 20:48 < mmcgrath> ricky: FWIW, I still use zabbix, it's sitting on my desktop now and beeps at me :) 20:48 < dgilmore> and rather than fix it you disabled it 20:48 < dgilmore> but i could be remebering wrong 20:48 < mmcgrath> that could be 20:48 < ricky> Sounds like me alright 20:48 < mmcgrath> well no rush to make a decision today. nagios alerts are still going out and paged alerts are going out 20:49 < mmcgrath> and at the moment, zabbix does have trending of some of the important things we need 20:49 < mmcgrath> like /mnt/koji usage 20:49 < ricky> The question is whether we should be spending time on improving nagios monitoring 20:49 < ricky> Like adding checks and keeping it in sync with hosts. That stuff has mostly stagnated since we looked at zabbix 20:49 < mmcgrath> yeah 20:49 < mmcgrath> I'm not sure how out of sync they are. 20:49 < mmcgrath> and zabbix has a lot of stuff nagios doesn't monitor. 20:49 < ricky> Is that a yeah to "we should be spending time on it" ? :-) 20:49 < mmcgrath> but that nagios doesn't really need to 20:50 < mmcgrath> just yeah, that stuff has stagnated 20:50 < mmcgrath> I'll take a look at things and try to get a better estimate of the work that needs to be done. 20:50 < dgilmore> ricky: i know we have hosts not in nagios 20:51 < mmcgrath> yeah 20:51 < mmcgrath> but most of our exernal service are still properly montiored by it 20:52 < mmcgrath> properly(ish) 20:52 < mmcgrath> ok, so more work to do, I'll get on that. 20:52 < mmcgrath> anyone have anything else? If not I'll open the floor 20:52 < smooge> all I want in a monitoring solution is something where adding a host to be monitorer can be done via puppet +files 20:53 < mmcgrath> yeah 20:53 < mmcgrath> ok 20:53 < ricky> Same here - the question seems to be what we want for a trending solution :-/ 20:53 < smooge> I hate clicking on things or depending on having to click on things. But open floor 20:53 < mmcgrath> #topic open floor 20:53 -!- zodbot changed the topic of #fedora-meeting to: open floor (Meeting topic: Fedora Infrastructure 2010-03-11) 20:53 < mmcgrath> ricky: I would love it if something like sar had the ability for arbitrary input :) 20:54 * gholms|work raises hand 20:54 < mmcgrath> gholms|work: yo 20:54 < gholms|work> Do you folks still plan on moving to Zenoss once 3.0 is released? 20:54 < mmcgrath> gholms|work: I don't think we ever planned on moving to Zenoss 20:54 < mmcgrath> so not to still, and no to zenoss 3.0 :) 20:54 < gholms|work> Heh, ok. 20:54 * gholms|work wonders where that idea came from... 20:54 < mmcgrath> I thought zenoss wasn't totally free? 20:55 < mmcgrath> I know it's not in Fedora yet, and that's a requisite 20:55 < gholms|work> It has a FOSS version. The big hangup was that it relied on bundled Python 2.4. 20:55 < mmcgrath> Ok, anyone have anything else to discuss? 20:56 < ricky> Just want to get this some visibility 20:56 < ricky> Jason Walsh has been working on poseidon, an OpenID provider writtin in pylons to replace the broken stuff in FAS 20:56 < ricky> So I'll be trying to get it setup with FAS auth on a publictest soon 20:57 < mmcgrath> ricky: excellent 20:57 < mmcgrath> happy to hear that one's been making progress. 20:57 < ricky> Which will hopefully be a good workout for the auth middleware in python-fedora :-) 20:57 < mmcgrath> ricky: are we the only users / potential users? 20:57 < ricky> Sorry, not sure what you mean 20:58 < mmcgrath> does anyone else use poseidon? 20:58 < ricky> Oh, no - it was started and written for this 20:58 < mmcgrath> k 20:58 < dgilmore> ricky: do you think we could have a way to use openid, present the cla and loow wiki edits etc 20:58 < dgilmore> and things like bodhi we could use openid sans cla 20:59 < dgilmore> for feedback 20:59 < ricky> Hm, that could get messy, as openids not linked to an existing FAS account could create all sorts of corner cases 20:59 < dgilmore> or am i all sorts of crazy 20:59 < mmcgrath> dgilmore: I think this is more of a provider 20:59 -!- mbonnet is now known as mbonnet_ 20:59 < mmcgrath> to accept openid from places would require a different sort of work 20:59 < dgilmore> mmcgrath: ok 20:59 < ricky> This is more of a "packagers can login to upstream bugzillas, etc. with their OpenID" sort of thing. 21:00 < ricky> (If those exist - or even just commenting on blogs) 21:00 < dgilmore> cool 21:00 * abadger1999 shows up in time for open mic^Wfloor 21:00 < dgilmore> just throwing my crazy ideas out there 21:00 < mmcgrath> yup yup 21:00 < mmcgrath> time to close the meeting, if no one has any objections we'll close in 30 21:01 * dgilmore wants to hear abadger1999 21:01 < ricky> ... sing :-) 21:01 < abadger1999> dgilmore: Interpretive dance :-) 21:01 < gholms|work> Over IRC?!? 21:01 < dgilmore> abadger1999: seen it already 21:01 < mmcgrath> we can talk about this next time or in #fedora-admin if we want, lets not hold everyone here :) 21:01 < dgilmore> mmcgrath: close it up 21:01 < mmcgrath> #endmeeting 21:01 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule 21:01 < zodbot> Meeting ended Thu Mar 11 21:01:31 2010 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . 21:01 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.html 21:01 < mmcgrath> thanks for coming everyone@ 21:01 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.txt 21:01 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2010-03-11/fedora-meeting.2010-03-11-20.02.log.html 21:01 < mmcgrath> !
Attachment:
pgp3goMYnFUe2.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
