20:00 < mmcgrath> #startmeeting Infrastructure 20:00 < zodbot> Meeting started Thu Mar 25 20:00:19 2010 UTC. The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00 < mmcgrath> Who's here? 20:00 * skvidal is here 20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic: Infrastructure) 20:00 * lmacken 20:00 * a-k is here 20:00 -!- djf_jeff [~jeff@xxxxxxxxxxxxx] has joined #fedora-meeting 20:01 -!- rwmjones [~rjones@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Ping timeout: 248 seconds 20:01 -!- inode0 [~inode0@fedora/inode0] has joined #fedora-meeting 20:02 < mmcgrath> #topic Fedora 13 beta 20:02 -!- zodbot changed the topic of #fedora-meeting to: Fedora 13 beta (Meeting topic: Infrastructure) 20:02 < mmcgrath> hah 20:02 < mmcgrath> Lets get started 20:02 < mmcgrath> https://fedorahosted.org/fedora-infrastructure/report/9 20:03 < mmcgrath> .ticket 2058 20:03 < zodbot> mmcgrath: #2058 (Verify Mirror Space) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2058 20:03 < mmcgrath> I'll get this one 20:03 < mmcgrath> .ticket 2059 20:03 < zodbot> mmcgrath: #2059 (Release Day ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2059 20:03 < mmcgrath> This one's just a tracker ticket. I'll take it too 20:03 < mmcgrath> .ticket 2060 20:03 < zodbot> mmcgrath: #2060 (Verify releng permissions) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2060 20:03 < mmcgrath> smooge: want to get that one again? 20:04 < mmcgrath> we'll come back to that. 20:04 < mmcgrath> .tiny 2061 20:04 < zodbot> mmcgrath: Error: '2061' is not a valid url. 20:05 < mmcgrath> .ticket 2061 20:05 < mmcgrath> sorry :/ 20:05 < zodbot> mmcgrath: #2061 (MM redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2061 20:05 < mmcgrath> mdomsch usually gets this one (I believe it's automated now and just requires verification) 20:05 < mmcgrath> .ticket 2062 20:05 < zodbot> mmcgrath: #2062 (Infrastructure Change Freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2062 20:05 < mmcgrath> I'll get that, we are frozen. 20:05 < smooge> morning 20:05 < smooge> sorry got stuck on phone 20:05 < mmcgrath> smooge: hey, want to get 2060? 20:05 < smooge> yes 20:05 < mmcgrath> k 20:05 < mmcgrath> and the last ticket 20:06 < mmcgrath> .ticket 2063 doesn't need to be done until after the launch 20:06 < smooge> could I get 2058 .. I find them related 20:06 < zodbot> mmcgrath: Error: "2063 doesn't need to be done until after the launch" is not a valid integer. 20:06 < smooge> hehheeh 20:06 -!- djf_jeff [~jeff@xxxxxxxxxxxxx] has quit Quit: I quit 20:06 < mmcgrath> zodbot: you are testing me! 20:06 < mmcgrath> ok, anyone have any questions or comments related to the release? 20:06 < mmcgrath> Oxf13: you around? 20:07 < Oxf13> I am 20:07 < mmcgrath> what are our odds of slipping at the moment? 20:07 < Oxf13> I'd put it at 50% chance 20:07 -!- biertie [~bert@fedora/biertie] has joined #fedora-meeting 20:07 < Oxf13> There is one blocker we're worried about, but we have a patch in hand, it just needs testing, then I can make the RC 20:07 < skvidal> mmcgrath: so - when it's a 50% chance of rain - you carry an umbrella :) 20:07 < mmcgrath> cool. 20:07 < Oxf13> we have a compressed amount of time to test the RC 20:08 < Oxf13> and not really any time to fix anything that's wrong with the RC and validate a second RC before the go / no go time 20:08 < mmcgrath> Oxf13: I'll work with you later today or tomorrow to verify mirror space, we expecting this to be the same size(ish) as the alpha? 20:08 < Oxf13> yes 20:08 < mmcgrath> k, sounds good. 20:08 < mmcgrath> If no one has anything else, I'll move on? 20:08 * smooge remembers the days of having 8 or 9 RC's 20:09 < smooge> nothing else 20:09 < mmcgrath> #topic func updates 20:09 -!- zodbot changed the topic of #fedora-meeting to: func updates (Meeting topic: Infrastructure) 20:09 < mmcgrath> So after some coding and some testing, the func updates before the freeze went pretty well I thought. 20:09 < skvidal> does anyone else want to work on that project? 20:09 < mmcgrath> still a few kinks to work out but it was much easier then our current method and required much less attention. 20:09 < mmcgrath> skvidal: as in you're done with it or want some help? 20:10 < skvidal> want some help 20:10 < skvidal> I got asked to work on something else this week 20:10 < mmcgrath> skvidal: I have some cycles during the freeze. though I can't promise I won't make things worse :) 20:10 < skvidal> and that's been taking my focus 20:11 < skvidal> so it's not dropped 20:11 < mmcgrath> skvidal: well I'm sure I'll be pinging you soon(ish) 20:11 < skvidal> but I won't be able to spend as much time on it until I get the mock vm stuff out 20:11 < mmcgrath> <nod> 20:11 < mmcgrath> anyone else have questions or comments on that? 20:11 < skvidal> the func+yum thing is lightweight 20:11 < skvidal> and entry-level easy to work on 20:11 < skvidal> http://fedorapeople.org/gitweb/skvidal/func-yum.git 20:12 < skvidal> lots of easy wins 20:12 < mmcgrath> skvidal: thanks 20:12 < mmcgrath> Ok, next topic 20:12 < mmcgrath> #topic Collectd 20:12 -!- zodbot changed the topic of #fedora-meeting to: Collectd (Meeting topic: Infrastructure) 20:12 < mmcgrath> So we've been using collectd for a bit now. 20:12 < mmcgrath> what do people think? 20:12 < abadger1999> skvidal: Thanks for getting func working well again. 20:12 < skvidal> abadger1999: so much to do to make things 'well' 20:12 < smooge> skvidal, I would like to help 20:12 < skvidal> but it is unbroken 20:13 < abadger1999> :-) 20:13 < smooge> my python is broken, but I really want to help 20:13 < smooge> sorry meant help on func 20:13 < smooge> collectd I have found useful 20:13 < abadger1999> mmcgrath: It's helped us fix something already. It's generally good. 20:13 < smooge> looking at app04 I can see where it is heavily running into some issues 20:13 < mmcgrath> smooge: me too, we've already found several problems just by having it in place. 20:14 < skvidal> what did collectd help you fix? 20:14 -!- rwmjones [~rjones@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:14 < mmcgrath> skvidal: it helpped us find the outage blips as being realted to db2. 20:14 < skvidal> ah 20:14 < skvidal> cool 20:14 -!- pcalarco_afk [~pcalarco@fedora/pcalarco] has quit Quit: Ex-Chat 20:15 < mmcgrath> other tools could have found it, but just the way we have it setup right now (every 10s) allowed us to see that load spike in such a short window was related to disk, and even more then that disk writes. 20:15 < mmcgrath> which got us looking. 20:15 < mmcgrath> and while it's not totally fixed I do think we're in better shape. I think we just need to adjust our backup system a bit. 20:15 < mmcgrath> but that'll be a post-freeze thing. 20:15 < mmcgrath> so here's the only got'cha with collectd. 20:15 < skvidal> its a massive suck? 20:15 < mmcgrath> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=log01&plugin=disk×pan=604800&action=show_selection&ok_button=OK 20:15 < mmcgrath> heh 20:16 < mmcgrath> it's the disk IO required to do rrd files. 20:16 < mmcgrath> there's lots of tricks we can (and do) use to fix that. 20:16 < mmcgrath> but as we grow, it's something to watch. 20:16 < mmcgrath> you'll notice that on the 19th I figured out that automatically polling every tcp port in use and recording that info was too expensive for us :) 20:16 < mmcgrath> duh 20:16 < mmcgrath> but yeah, something to watch. 20:17 < mmcgrath> there's also non-rrdtool collection methods we can use if we really need to that would also be useful 20:17 < mmcgrath> anywho, anyone have any questions on that? 20:17 < mmcgrath> I've been slowly adding more useful stuff 20:17 < mmcgrath> like - 20:17 < mmcgrath> .tiny https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db1&plugin=mysql×pan=86400&action=show_selection&ok_button=OK 20:17 < zodbot> mmcgrath: http://tinyurl.com/ye6nrvl 20:18 < mmcgrath> anywho, no more questions there so that's good. 20:18 < mmcgrath> #topic Monitoring 20:18 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic: Infrastructure) 20:18 < smooge> mmcgrath, we did rrdtools in a ram drive 20:18 < mmcgrath> smooge: yeah that's basically what some of the tunables in the rrdtool plugin do. 20:18 < mmcgrath> Ok, so we're basically back to nagios and collectd. 20:19 < smooge> yeah.. we set it up like /var/spool/mqueue/t and had a 2 GB partition for it .. 20:19 < mmcgrath> hydh has been working on some stuff but I know he's been busy 20:19 < smooge> I want to say our community help on nagios has been cool 20:19 < smooge> and great 20:19 < mmcgrath> I'd like to get some basic event handlers finalized as well as proper deps in place. 20:19 < mmcgrath> smooge: indeed 20:20 < mmcgrath> anyone have any questions about what we're up to in nagios and where we're headed? 20:20 < mmcgrath> alrighty, well with that 20:20 < mmcgrath> #topic Open Floor 20:20 -!- zodbot changed the topic of #fedora-meeting to: Open Floor (Meeting topic: Infrastructure) 20:20 < mmcgrath> anyone have anything they'd like to discuss? 20:20 < mmcgrath> a-k: anything new on search engines? 20:20 -!- XulLunch is now known as XulWork 20:21 < a-k> I'm still looking at mnoGoSearch with PostgreSQL 20:21 < a-k> I haven't had a chance to try crawling with it yet, but if it goes well I'll put it in pub test next week 20:21 < a-k> BTW is there a preference for MySQL vs PostgreSQL? I know/think we have them both around.... 20:21 < dgilmore> a-k: no preference 20:21 < a-k> I'm okay with that. That's about it for now. 20:22 < mmcgrath> anyone have anything else they'd like to discuss/ 20:22 < gholms> Random question? 20:22 < gholms> Did you folks notice the collectd server-side daemon putting a lot of load on the machine? I ask because that's what I experienced at $dayjob. 20:23 < mmcgrath> gholms: yeah, we fixed it with the suggestions here... 20:23 * mmcgrath gets link 20:23 < mmcgrath> http://collectd.org/wiki/index.php/Inside_the_RRDtool_plugin 20:24 < gholms> Ooh, that looks useful. Thanks. 20:24 < mmcgrath> gholms: yup yup 20:24 < smooge> 1) going to work on log reviews over freeze. Now that we have over 50% free logs I was going to see what I could get out of it daily. 20:24 < smooge> I think someone else was working on this earlier so will hook up with them and see where it goes 20:24 < mmcgrath> smooge: cool 20:24 < mmcgrath> yeah someone was but I don't know the status 20:25 < smooge> then I am pretty much building my home/slicehost network to 'clone' F-I so I can test stuff here a bit better. 20:25 < smooge> my goal will be to see how far I can take epylog before it screams in terror at our data 20:25 < mmcgrath> smooge: sounds good, let us know if you need anything 20:25 < mmcgrath> well me 20:25 < mmcgrath> :) 20:26 -!- fab_ [~bellet@xxxxxxxxxxx] has quit Ping timeout: 248 seconds 20:26 < mmcgrath> ok, well with that I'll close the meeting in 30 20:26 < mmcgrath> 15 20:26 < mmcgrath> #endmeeting 20:26 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule 20:26 < zodbot> Meeting ended Thu Mar 25 20:26:56 2010 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . 20:27 < mmcgrath> thanks for coming everyone! 20:27 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.html 20:27 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.txt 20:27 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2010-03-25/fedora-meeting.2010-03-25-20.00.log.html
Attachment:
pgpHqwnrdz86I.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
