2008-07-31T15:59:39 *** mmcgrath changes topic to "Infrastructure -- Who's here?" 2008-07-31T15:59:43 <mmcgrath> Who's around? 2008-07-31T15:59:45 <ianweller> there are too many meetings on wednesdays 2008-07-31T15:59:50 <G> me me me! 2008-07-31T15:59:50 * ianweller 2008-07-31T15:59:58 * wakko6661 is here, but waiting for his other session to time out before reclaiming his /nick. 2008-07-31T16:00:18 <londo> here 2008-07-31T16:00:20 <ianweller> wakko6661: /msg nickserv ghost 2008-07-31T16:00:29 <fchiulli> fchiulli here 2008-07-31T16:00:50 *** wakko6661 is now known as wakko666 2008-07-31T16:00:52 <mmcgrath> ianweller: today's thursday :) 2008-07-31T16:00:57 <wakko666> ianweller: thanks. :) 2008-07-31T16:01:04 <ianweller> mmcgrath: today too! 2008-07-31T16:01:17 <ianweller> why can i never remember what day of the week it is 2008-07-31T16:01:24 <ianweller> this is going to be a problem when school rolls around :( 2008-07-31T16:01:27 <mmcgrath> Ok, lets get started 2008-07-31T16:01:31 <mmcgrath> ianweller: because schools out ;-) 2008-07-31T16:01:33 <mmcgrath> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority 2008-07-31T16:01:34 <zodbot> <http://tinyurl.com/2hyyz6> (at fedorahosted.org) 2008-07-31T16:01:40 <mmcgrath> .ticket 671 2008-07-31T16:01:45 <mmcgrath> anyone from in.fedoraproject.org around? 2008-07-31T16:01:49 <mmcgrath> grr zodbot 2008-07-31T16:01:51 <SmootherFrOgZ> hello 2008-07-31T16:01:54 <mmcgrath> https://fedorahosted.org/fedora-infrastructure/ticket/671 2008-07-31T16:02:13 <ianweller> mmcgrath: since you're not using it anyway you can comment out the python-fedora fetch line in __init__ i think 2008-07-31T16:02:45 <kschreyack> present 2008-07-31T16:02:48 *** ianweller changes topic to "Infrastructure -- Tickets" 2008-07-31T16:03:01 <mmcgrath> .load Fedora 2008-07-31T16:03:01 <zodbot> mmcgrath: The operation succeeded. 2008-07-31T16:03:11 <mmcgrath> Ok, I'm removing ticket 671 if no one is around. 2008-07-31T16:03:13 *** mcgarnicle has joined #fedora-meeting 2008-07-31T16:03:21 <mmcgrath> err removing the "Meeting" flag 2008-07-31T16:03:27 <abadger1999> BTW, that method interacts truly horribly with our privacy stuff... going to have to figure something else out for it. 2008-07-31T16:03:28 <mmcgrath> we're skipping 732 for now (end of meeting) 2008-07-31T16:03:29 <G> skip it 2008-07-31T16:03:34 <mmcgrath> abadger1999: solid 2008-07-31T16:03:37 <mmcgrath> .ticket 395 2008-07-31T16:03:38 <zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/395 2008-07-31T16:03:39 <zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - http://tinyurl.com/5qb3gv 2008-07-31T16:03:43 <mmcgrath> jcollie: any update? 2008-07-31T16:04:07 <yingbull> mmcgrath: I'm here too, btw, just missed the start. 2008-07-31T16:05:10 <ivazquez> As am I. 2008-07-31T16:05:13 <G> jcollie said he was busy today 2008-07-31T16:05:15 *** buggbot has quit IRC 2008-07-31T16:05:16 <mmcgrath> yingbull: yo 2008-07-31T16:05:18 <mmcgrath> thats fine 2008-07-31T16:05:21 <mmcgrath> we'll skip that ticket for now. 2008-07-31T16:05:28 <mmcgrath> .ticket 446 2008-07-31T16:05:29 <zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/446 2008-07-31T16:05:30 *** buggbot has joined #fedora-meeting 2008-07-31T16:05:31 <zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - http://tinyurl.com/3w4uwn 2008-07-31T16:05:31 <ianweller> daaang, buggbot's having issues 2008-07-31T16:05:33 <mmcgrath> dgilmore: ping? 2008-07-31T16:05:48 *** ricky has joined #fedora-meeting 2008-07-31T16:05:55 * ricky is here 2008-07-31T16:05:58 <mmcgrath> ricky in the house 2008-07-31T16:06:07 <ricky> (ISP had a loooong outage) 2008-07-31T16:06:11 <ianweller> ricky: ouch 2008-07-31T16:06:51 *** roguedaemon has left #fedora-meeting 2008-07-31T16:07:18 <mmcgrath> ok, we'll skip that ticket as well. 2008-07-31T16:07:21 <mmcgrath> .ticket 576 2008-07-31T16:07:22 <zodbot> mmcgrath: #576 (Infrastructure Contact Information) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/576 2008-07-31T16:07:24 <zodbot> mmcgrath: #576 (Infrastructure Contact Information) - Fedora Infrastructure - Trac - http://tinyurl.com/4kevcb 2008-07-31T16:07:42 <G> mmcgrath: I seem to remember you having a proposal for that which sounds reasonable 2008-07-31T16:07:51 <mmcgrath> G: so should I just setup a simple script + encfs to get that share up? Its handy because it allows for rsync local copies as well. 2008-07-31T16:07:59 <mmcgrath> G: the only problem I haven't figured out is... shared password. 2008-07-31T16:08:04 <mmcgrath> but thats not terrible. 2008-07-31T16:08:25 <mmcgrath> we can try it out and see what we think, if its horrible we can always change it :) 2008-07-31T16:08:35 <G> yeah, I think thats the best option 2008-07-31T16:08:37 <ricky> Cool 2008-07-31T16:08:38 <mmcgrath> Ok, so thats it for the tickets. I had a couple of other things to dicsuss before going back to 732 2008-07-31T16:08:46 *** mmcgrath changes topic to "Infrastructure -- Me gone (again)" 2008-07-31T16:08:57 <mmcgrath> So I'll be gone... again :) On Monday and Tuesday. 2008-07-31T16:09:04 <ricky> Heh. 2008-07-31T16:09:06 <mmcgrath> that should be my last vacation of the year so woo hoo 2008-07-31T16:09:07 <G> mmcgrath: where is thou going? :) 2008-07-31T16:09:07 <kschreyack> nice 2008-07-31T16:09:23 <ianweller> mmcgrath: i don't think i've ever seen anyone happy about not having any more vacations 2008-07-31T16:09:24 <mmcgrath> G: http://www.cedarpoint.com/ 2008-07-31T16:09:35 <mmcgrath> ianweller: its hard to get stuff done when you're camping :) 2008-07-31T16:09:44 <ricky> Ooh, fun 2008-07-31T16:09:58 <ianweller> ah, fedora. the job everyone (literally) lives to have 2008-07-31T16:09:58 <G> mmcgrath: ooh nice 2008-07-31T16:10:02 <SmootherFrOgZ> he he 2008-07-31T16:10:04 <mmcgrath> Ok, so thats that. 2008-07-31T16:10:10 *** mmcgrath changes topic to "Infrastructure -- Outage last night" 2008-07-31T16:10:16 <mmcgrath> really, wtf was that all about? 2008-07-31T16:10:22 <ricky> What about the network switch? 2008-07-31T16:10:23 <mmcgrath> Its the first time thats ever happened to us. 2008-07-31T16:10:27 <ricky> As in, does that happen while you're here? 2008-07-31T16:10:31 <mmcgrath> ricky: I haven't heard any updates on that 2008-07-31T16:10:44 <ricky> Ah, OK. 2008-07-31T16:10:46 * mmcgrath summons the notting 2008-07-31T16:11:05 <fchiulli> What was the cause of the problem? 2008-07-31T16:11:20 <mmcgrath> fchiulli: so here's what we know. 2008-07-31T16:11:25 <mmcgrath> the netapp crashed. 2008-07-31T16:11:27 <mmcgrath> not sure why 2008-07-31T16:11:33 <mmcgrath> they're still looking into it. 2008-07-31T16:11:41 <mmcgrath> but I do know that our current iptables rules got in the way of the fix. 2008-07-31T16:11:51 <mmcgrath> right now we can't mount the netapp with iptables enabled. 2008-07-31T16:11:57 <mmcgrath> after the mount is done, we can re-enable iptables. 2008-07-31T16:12:05 <mmcgrath> I looked briefly this morning but haven't figured out wtf is going on. 2008-07-31T16:12:08 <mmcgrath> I'll look more this afternoon. 2008-07-31T16:12:27 <G> and as a result it brought down all the services that had data on the netapp, i.e. db2 - fas and all that depends on it, and non-cached wiki data 2008-07-31T16:12:28 <ricky> At least, we can hopefully narrow down the tables rule 2008-07-31T16:12:34 <ricky> Did it use to work with iptables on before? 2008-07-31T16:12:36 <mmcgrath> correct. 2008-07-31T16:12:41 <mmcgrath> ricky: yeah 2008-07-31T16:12:54 <ricky> All right, so it's something we changed :-) 2008-07-31T16:12:57 <mmcgrath> I made the last change that I could find and it seemed pretty harmless. 2008-07-31T16:12:59 <londo> do the rpc services have fixed ports in the netapp? 2008-07-31T16:13:03 <mmcgrath> ricky: or its related to the rhel 5.2 update. 2008-07-31T16:13:08 <ricky> Ah 2008-07-31T16:13:09 <mmcgrath> londo: not sure. 2008-07-31T16:13:19 *** buggbot has quit IRC 2008-07-31T16:13:30 *** buggbot has joined #fedora-meeting 2008-07-31T16:13:44 <G> mmcgrath: maybe we need to say "pretty pretty please can you update our firmware" :) 2008-07-31T16:13:46 <ricky> Does NFS require any incoming connections? 2008-07-31T16:14:03 <mmcgrath> G: a good possibility. 2008-07-31T16:14:13 <mmcgrath> ricky: as in, does the client need to send something to the server? 2008-07-31T16:14:28 <ricky> Yeah. 2008-07-31T16:14:29 <ricky> Er. 2008-07-31T16:14:33 <ricky> The other way around 2008-07-31T16:14:43 <mmcgrath> for nfslock, I believe so yes. 2008-07-31T16:14:45 <mmcgrath> its udp based. 2008-07-31T16:14:48 <ricky> Ah 2008-07-31T16:14:55 <mmcgrath> its just something we need to look at. 2008-07-31T16:14:59 <mmcgrath> moving on... 2008-07-31T16:15:09 *** mmcgrath changes topic to "Infrastructure -- Switch upgrade." 2008-07-31T16:15:15 <mmcgrath> well, we're supposed to have a switch upgraded this week or next. 2008-07-31T16:15:24 <mmcgrath> not heard any ETA or anything about the nature of the upgrade, nothing. 2008-07-31T16:15:39 * mmcgrath sends an email 2008-07-31T16:15:48 <G> heh, classic! 2008-07-31T16:15:54 <mmcgrath> email sent. 2008-07-31T16:16:05 <mmcgrath> next 2008-07-31T16:16:10 *** mmcgrath changes topic to "Infrastructure -- Torrent" 2008-07-31T16:16:15 *** wakko666 has quit IRC 2008-07-31T16:16:17 <mmcgrath> Soooooooo. Server Beach isn't going to work out for Torrent. 2008-07-31T16:16:18 <ricky> Boo serverbeach :-( 2008-07-31T16:16:23 <mmcgrath> at least we don't think so. 2008-07-31T16:16:23 *** wakko666 has joined #fedora-meeting 2008-07-31T16:16:32 <mmcgrath> The last request I sent hasn't been answered yet so we're waiting to see. 2008-07-31T16:16:41 <mmcgrath> but there's some port blockage issues that we're seeing. 2008-07-31T16:16:41 <G> mmcgrath: so whats the current problems with SB? 2008-07-31T16:16:49 <ricky> G: They block port 6969 2008-07-31T16:16:57 <mmcgrath> G: nmap -p 6969 serverbeach1.fedoraproject.org 2008-07-31T16:16:59 <ianweller> lol w h y 2008-07-31T16:17:04 <G> 6969 = bittorrent right? 2008-07-31T16:17:11 <ianweller> can we ask them to disable that? 2008-07-31T16:17:14 <ricky> Because bittorrents are always illegal, of course. 2008-07-31T16:17:14 <mmcgrath> ianweller: ill-informed network staff I suspect. 2008-07-31T16:17:17 <mmcgrath> ianweller: we did 2008-07-31T16:17:36 <mmcgrath> I'm hoping they will move us to a different network. 2008-07-31T16:17:43 <ianweller> can we set up a gateway that funnels traffic through something if it doesn't work out? 2008-07-31T16:17:46 <mmcgrath> But there's problems.... 2008-07-31T16:17:56 <mmcgrath> 1) our current torrent server died earlier this week. Its REALLY old. 2008-07-31T16:18:02 <mmcgrath> and will likely be down for the count soon. 2008-07-31T16:18:04 <ricky> Wha? It did? 2008-07-31T16:18:05 <ricky> :-( 2008-07-31T16:18:06 <G> mmcgrath: well to be honest, bit torrent can be a bit of a pain, and it might not be sb's fault, it could be the datacenter's fault 2008-07-31T16:18:26 <mmcgrath> G: its actually their parent companies fault Peer 1. Policy of the edge router 2008-07-31T16:18:42 <mmcgrath> so in the meantime seth's been hard at work and we're keeping a backup of torrent on fedorapeople. 2008-07-31T16:18:45 <skvidal> ricky: not died, dead, but it oopsed in a bad place 2008-07-31T16:18:51 <ricky> Ah. 2008-07-31T16:19:02 <skvidal> and we have another copy of the data on sb1, too 2008-07-31T16:19:05 <G> yeah exactly, AtlantaNAP filter IRC at the datacenter level 2008-07-31T16:19:25 *** sharkcz has quit IRC 2008-07-31T16:19:47 <mmcgrath> so I'm just keeping everyone in the know on that. Like I said, its not a complete no at this point, they're still looking into it and we thank them for that and sponsoring our project. 2008-07-31T16:20:03 <skvidal> does anyone here have any contacts at any other hosting providers? 2008-07-31T16:20:08 *** xaeth has joined #Fedora-meeting 2008-07-31T16:20:34 <G> skvidal: not really, but... 2008-07-31T16:20:48 <mmcgrath> I'm going to open the floor for a few minutes, then move to ticket 732. 2008-07-31T16:20:54 <mmcgrath> G: ? 2008-07-31T16:20:58 <G> I've seen people from TP (The Planet) hanging around in -admin every now and then 2008-07-31T16:21:27 <mmcgrath> might be worth a cattle call, see who answers. 2008-07-31T16:21:30 <mmcgrath> anyone have anything else on that? 2008-07-31T16:21:38 <G> They own what used to be Ev1 etc, and I know Linode have quite a few servers at TP and seem happy 2008-07-31T16:22:11 <ianweller> is it possible to load balance a bt server or is that crazy talk 2008-07-31T16:22:30 <ricky> Is there a need to? 2008-07-31T16:22:38 * ianweller is just wondering 2008-07-31T16:22:39 * ricky has no idea how intensive bt stuff is 2008-07-31T16:22:41 <ianweller> i don't see a need. 2008-07-31T16:22:43 <ianweller> yet, at least 2008-07-31T16:22:52 <skvidal> ianweller: no real need 2008-07-31T16:23:08 <mmcgrath> ianweller: thats possible yet, not much in our use case right now since we're having problems finding a single place to host a torrent server, much less multiple torrent servers :) 2008-07-31T16:23:15 <mmcgrath> err possible yes, 2008-07-31T16:23:15 <ianweller> i'm just thinking, an automatic way to switch to a backup tracker in case one goes *splat* 2008-07-31T16:23:29 <skvidal> ianweller: we'd need to swap out dns entry 2008-07-31T16:23:30 <ricky> ianweller: We might do it with DNS 2008-07-31T16:23:36 <SmootherFrOgZ> skvidal: i suppose you mean *us* hosting providers ? 2008-07-31T16:23:50 <skvidal> SmootherFrOgZ: anyone, really 2008-07-31T16:24:01 <mmcgrath> SmootherFrOgZ: doesn't have to be but it'd have to be fast enough for releng to copy large files to it. 2008-07-31T16:24:32 <skvidal> very very large files 2008-07-31T16:24:38 <SmootherFrOgZ> well, i'll have a talk with our hosting section on tommorow then 2008-07-31T16:25:25 <ricky> Cool 2008-07-31T16:25:30 <ricky> Just curious: Where are you? 2008-07-31T16:25:33 <G> I think really, if we were able to do GeoDNS etc a proxy server in the APAC area might be handy too :) 2008-07-31T16:26:01 <mmcgrath> SmootherFrOgZ: thanks, even if not for torrent we can use another proxy / cache server there. 2008-07-31T16:26:09 *** fchiulli_ has joined #fedora-meeting 2008-07-31T16:26:17 <SmootherFrOgZ> k 2008-07-31T16:26:43 *** cassmodiah has quit IRC 2008-07-31T16:26:51 <mmcgrath> Ok, moving on 2008-07-31T16:26:59 *** mmcgrath changes topic to "Infrastructure -- Open Floor (before monitoring stuff)" 2008-07-31T16:27:03 <mmcgrath> anyone have anything to discuss? 2008-07-31T16:27:09 <ianweller> wiki slowness? 2008-07-31T16:27:23 *** fchiulli has quit IRC 2008-07-31T16:27:25 <ricky> I'll be looking into alternative session storage son 2008-07-31T16:27:26 <ricky> **soon 2008-07-31T16:27:27 <ianweller> gregdek had some issues with the wiki during our marketing meeting an hour ago, apparently session problems again 2008-07-31T16:27:30 <ianweller> ricky: sweet 2008-07-31T16:27:35 <ricky> Namely, memcached 2008-07-31T16:27:35 *** buggbot has quit IRC 2008-07-31T16:28:03 <mmcgrath> ricky: yeah I ment to ask you and G if anything changed recently. 2008-07-31T16:28:07 *** buggbot has joined #fedora-meeting 2008-07-31T16:28:09 *** fchiulli_ is now known as fchiulli 2008-07-31T16:28:10 <mmcgrath> we've got a bunch of empty sessions again. I'd fixed that a while back. 2008-07-31T16:28:19 * dgilmore is here 2008-07-31T16:28:35 <G> mmcgrath: I hadn't changed anything with production recently 2008-07-31T16:29:05 <G> mmcgrath: might be something to do with the netapp weirdness etc 2008-07-31T16:29:10 <mmcgrath> possibly. 2008-07-31T16:29:14 <mmcgrath> something we'll want to look at. 2008-07-31T16:29:40 <mmcgrath> Ok, if there's nothing else, on to the ticket. 2008-07-31T16:29:54 <mmcgrath> .ticket 732 2008-07-31T16:29:55 <zodbot> mmcgrath: #732 (Investigate New System Monitoring Methods) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/732 2008-07-31T16:29:56 <zodbot> mmcgrath: #732 (Investigate New System Monitoring Methods) - Fedora Infrastructure - Trac - http://tinyurl.com/69whk6 2008-07-31T16:29:58 <G> okay, so I think this is my part right? 2008-07-31T16:30:01 <mmcgrath> G's got lead on this so have at it. 2008-07-31T16:30:19 *** G changes topic to "Infrastructure -- Server Monitoring" 2008-07-31T16:31:11 <G> So to recap, the last few days we have been talking about the possibility of switching from Nagios to another solution, after a bit of poking around, it seems Zabbix and Zenoss are the better of the others 2008-07-31T16:31:37 * dgilmore stays out of this 2008-07-31T16:31:47 <skvidal> not zope. please 2008-07-31T16:32:05 <G> As zenoss suffers from requiring Python 2.4 (zope) and we can't guarrantee that it'll work with RHEL6, it seems Zabbix is our primary option 2008-07-31T16:32:30 <G> skvidal: I was getting to that :) 2008-07-31T16:32:39 <mmcgrath> also zenoss isn't in Fedora. 2008-07-31T16:32:40 <fchiulli> Zabbix uses PHP correct? 2008-07-31T16:32:47 <G> fchiulli: it's a mix 2008-07-31T16:33:01 <fchiulli> G: of what? 2008-07-31T16:33:05 <G> The front end is fedora, but everything else I think is compiled C 2008-07-31T16:33:07 <ricky> PHP and c, it seems 2008-07-31T16:33:08 <G> errr 2008-07-31T16:33:12 <G> the Front end is PHP 2008-07-31T16:33:14 <ricky> The PHP looks very C-like too :-) 2008-07-31T16:33:48 <G> fchiulli: it's split up into 3 components, Frontend, Server and Agents 2008-07-31T16:34:46 <G> wakko666 and jcollie have put their hands up as DAYJOB users and have offered to help us out if we wish to implement it 2008-07-31T16:35:02 *** stickster is now known as stickster_mtg 2008-07-31T16:35:47 <mmcgrath> G: I got zabbix setup at home. Its got a little learning curve. I'd say not as hard as nagios though. 2008-07-31T16:36:05 <mmcgrath> Its important to get the different terms known. Event, Action, Trigger, Application, etc. 2008-07-31T16:36:19 <G> Mike proposed that it might be an idea to setup an instance alongside nagios for now, monitoring 10 or so machines in our infrastructure (I can see koji, a builder, a couple of app servers etc as primary candidates) 2008-07-31T16:36:23 <ricky> Can we get a quick rundown of this stuff if we're going to start testing in production at some point? 2008-07-31T16:37:01 <mmcgrath> ricky: well, unlike nagios, zabbix does trending so we'll be monitoring the stuff we are now, and then some. 2008-07-31T16:37:07 <mmcgrath> but we won't, for example, get notified about all of it. 2008-07-31T16:37:33 <G> okay the important bits are: 2008-07-31T16:38:01 <G> a trigger is set-off when an item been monitored say apache, goes down 2008-07-31T16:38:12 <G> in that situation the trigger is said to be TRUE 2008-07-31T16:38:56 <mmcgrath> right, in nagios I always tended to think of it as monitoring when the host was up and notifying us when its down. zabbix tends to only think of things as being down. 2008-07-31T16:39:02 <mmcgrath> IE, False == good. True == bad :) 2008-07-31T16:39:11 *** Sonar_Guy has joined #fedora-meeting 2008-07-31T16:39:23 <G> an event is a nice way of interpreting the triggers and a combination of triggers 2008-07-31T16:39:37 <ianweller> mmcgrath: the truth hurts 2008-07-31T16:40:10 <G> an action is performed when a trigger/action returns TRUE/ON and is something like, remote commands, e-mails, jabber messages etc 2008-07-31T16:40:17 *** buggbot has quit IRC 2008-07-31T16:40:46 *** buggbot has joined #fedora-meeting 2008-07-31T16:40:57 <fchiulli> G: Can an action perform multiple tasks? 2008-07-31T16:41:05 <G> fchiulli: yeah 2008-07-31T16:41:16 <mmcgrath> fchiulli: an action can be a script so it could do whatever you want. 2008-07-31T16:41:40 <G> http://publictest3.fedoraproject.org/zabbix/events.php for instance shows us that pt2 got rebooted recently 2008-07-31T16:42:03 <mmcgrath> The manual is actually quite good. if you're going to be using zabbix you need to give it a look because for items and triggers you need to know the pretty small zabbix language. 2008-07-31T16:42:14 <G> mmcgrath: yeah 2008-07-31T16:42:15 <mmcgrath> So, for example, {desktop:vm.memory.size[free].last(0)}<10000 2008-07-31T16:42:26 <ricky> +1 to the documentation 2008-07-31T16:42:42 <G> hmmm and pt2 was rebooted recently ): 2008-07-31T16:42:44 <G> *:) 2008-07-31T16:42:49 <ianweller> that is very interesting. 2008-07-31T16:42:52 <mmcgrath> thats on my zabbix machine at home, its for my desktop machine, its looking at vm free, its using the current value (of last(0)) and is checking that its not less then 10000 2008-07-31T16:43:07 <ricky> So looking at http://publictest3.fedoraproject.org/zabbix/actionconf.php, right now, there the only thing that has notifications now is apache on pt9? 2008-07-31T16:43:25 <G> correct, and that only goes to me atm 2008-07-31T16:43:37 *** tc141516 has quit IRC 2008-07-31T16:43:41 <G> it's not actually properly setup, it was just the first thing I did when I first setup the server 2008-07-31T16:43:46 <fchiulli> mmcgrath: Is the time from the host (pt2) or the Zabbix server? 2008-07-31T16:43:59 *** Standby has joined #fedora-meeting 2008-07-31T16:43:59 <mmcgrath> fchiulli: no idea, G? 2008-07-31T16:44:00 <G> fchiulli: UTC 2008-07-31T16:44:02 <mcgarnicle> I'm just lurking here, but I'd like to point out Zenoss does have "stack" installers that include all deps, so maybe the python version isn't as be of an issue for the future 2008-07-31T16:44:21 <mmcgrath> mcgarnicle: cool, unfortunately for us its not in Fedora so we can't use it :( 2008-07-31T16:44:21 <ricky> mcgarnicle: We usually like to work with stuff that can be packaged 2008-07-31T16:44:25 <ricky> As in, that's a packaging no-no 2008-07-31T16:44:38 <wakko666> mcgarnicle: zenoss uses rpath, which makes it a huge pain to maintain 2008-07-31T16:44:45 <G> mcgarnicle: yeah, the funny thing is, I tried that and it blew up on me :) 2008-07-31T16:44:45 * ianweller sees a graphs tab 2008-07-31T16:44:59 <G> ianweller: thats the trending 2008-07-31T16:45:00 <mmcgrath> G: I even got a map setup today, took about 5 minutes for my first time through ;) 2008-07-31T16:45:08 <G> http://publictest3.fedoraproject.org/zabbix/charts.php?groupid=0&hostid=0&graphid=28 2008-07-31T16:45:09 <zodbot> <http://tinyurl.com/62t2ns> (at publictest3.fedoraproject.org) 2008-07-31T16:45:10 <ianweller> G: so nothing like cacti? 2008-07-31T16:45:35 <mmcgrath> ianweller: I wouldn't say nothing like cacti, but very different in a number of key architectural ways. 2008-07-31T16:45:41 <mmcgrath> 1) cacti uses rrd and zabbix does not 2008-07-31T16:45:47 <wakko666> ianweller: the graphs are IMO, better than cacti. 2008-07-31T16:45:48 <ianweller> i noticed that 2008-07-31T16:45:49 <G> imo it's better than cacti, it can graph anything that it monitors 2008-07-31T16:45:55 <mmcgrath> 2) adhoc graphs are much easier. 2008-07-31T16:45:57 <ianweller> can it to snmp? 2008-07-31T16:46:03 <G> yes it can 2008-07-31T16:46:03 <wakko666> ianweller: yup 2008-07-31T16:46:06 <ricky> In terms of "best practice," should actions be as fine-grained as possible? 2008-07-31T16:46:08 <mmcgrath> ianweller: yeah there's actually an snmp section in the manual but I havent' read that. 2008-07-31T16:46:11 <ianweller> rrdtool is a pain in the ass most times >.< 2008-07-31T16:46:27 <wakko666> i've done snmp stuff with zabbix against F5 LTMs and HP ProCurves 2008-07-31T16:46:37 <ricky> G: It might be a good idea to have a set of best practices for configuring this 2008-07-31T16:46:45 <mmcgrath> G: wakko666: one thing I wondered about is when would you use snmp vs just the agent? 2008-07-31T16:46:46 <ricky> Might be useful for csi, even :-) 2008-07-31T16:47:08 <wakko666> mmcgrath: my opinion is to use snmp when installing the agent isn't possible 2008-07-31T16:47:11 <G> ricky: yeah I will be 2008-07-31T16:47:20 <mmcgrath> ahh, ok. so the new switching hardware is not next week, sometime next month, he ordered it recently. 2008-07-31T16:47:21 <G> wakko666: I agree :) 2008-07-31T16:47:21 <wakko666> i.e appliances and such 2008-07-31T16:47:41 <ricky> Ah, OK 2008-07-31T16:47:42 *** tomspur has joined #fedora-meeting 2008-07-31T16:47:47 <G> anyway, to get this back on track for now 2008-07-31T16:48:06 <G> A) Are we happy to take this to the next level? 2008-07-31T16:48:29 <mmcgrath> A) yes. 2008-07-31T16:48:40 <G> set it up in parrallel with nagios on noc1 monitoring 10 hosts 2008-07-31T16:48:50 <mmcgrath> G: I figure certainly for this initial trial we'll have each person setup their own notifications. 2008-07-31T16:48:50 <G> okay 2008-07-31T16:49:04 <mmcgrath> I'm fine getting multiple pages and things, I'll be curious to see which ones we get first. 2008-07-31T16:49:13 <londo> the configuration is web based only? 2008-07-31T16:49:18 <wakko666> londo: correct 2008-07-31T16:49:19 <G> B) What hosts do we want to monitor 2008-07-31T16:49:20 <ricky> londo: Yes :-( 2008-07-31T16:49:49 <mmcgrath> there's an import export system so I suspect it could be scripted. 2008-07-31T16:49:49 <ricky> G: Curious: For everything listed under "Default Settings," did you configure those manually, or were they there already? 2008-07-31T16:50:01 <mmcgrath> londo: there's also a lot of auto detection auto discovery stuff. 2008-07-31T16:50:01 <wakko666> mmcgrath: correct. the import/export is in xml. 2008-07-31T16:50:06 <G> they were already there 2008-07-31T16:50:13 <ricky> All right, /me searches harder, then :-) 2008-07-31T16:50:31 <mmcgrath> ricky: yeah the setup I did at home had plenty of stuff already 2008-07-31T16:50:44 <G> btw, there is auditing, so we know who changes stuff, and I'll know who to chase with a 2x4 if they stuff it up :) 2008-07-31T16:50:49 <wakko666> i used the import/export heavily when extending the snmp checks to include hardware-specific MIBs 2008-07-31T16:51:10 <fchiulli> mmcgrath: Is there any plan to put the configs in puppet maybe using import/export? 2008-07-31T16:51:22 <G> I think thats an ugly method tbh 2008-07-31T16:51:25 <mmcgrath> fchiulli: naw, I don't think we really ahve that many configs. 2008-07-31T16:51:43 <mmcgrath> much of what zabbix does by default will be covered, the additional stuff we'll just do by hand and shouldn't be much. 2008-07-31T16:51:45 <wakko666> fchiulli: there's a limit to what is exported. it doesn't export everything, so if that's our restore plan, there are many things that would be lost 2008-07-31T16:51:51 <fchiulli> mmcgrath: But export for use to back it up. 2008-07-31T16:51:52 <ricky> If we test this in prod, it'll be a completely fresh install, correct? 2008-07-31T16:51:56 <G> ANYWAY, back to item B) 2008-07-31T16:52:04 <mmcgrath> fchiulli: ah, actually we'll just make db backups. 2008-07-31T16:52:17 <mmcgrath> G: hosts to monitor. I'd say at least a couple of services. 2008-07-31T16:52:18 <G> What hosts do we want to see monitored? 2008-07-31T16:52:22 <mmcgrath> hosted1, hosted2. 2008-07-31T16:52:25 <mmcgrath> app1-4 2008-07-31T16:52:28 <mmcgrath> bastion 2008-07-31T16:52:29 <ricky> nfs1? 2008-07-31T16:52:29 <G> koji1 2008-07-31T16:52:31 <mmcgrath> proxy1 2008-07-31T16:52:32 <mmcgrath> koji1 2008-07-31T16:52:34 <G> kojipkgs1 2008-07-31T16:52:42 <mmcgrath> yeah, I'd say any or all of those would be fine 2008-07-31T16:52:43 <mmcgrath> ricky: yeah nfs1 2008-07-31T16:52:44 <G> x86-1 and ppc3 maybe? 2008-07-31T16:53:00 <wakko666> do we want to do any website montioring as well as host monitoring? 2008-07-31T16:53:01 <ricky> Oh yeah, a builder would be good 2008-07-31T16:53:02 <mmcgrath> G: actually xenbuilder4 and ppc4. I think they're our busiest. 2008-07-31T16:53:08 <G> mmcgrath: k 2008-07-31T16:53:12 <mmcgrath> wakko666: yeah I'd like to. 2008-07-31T16:53:27 <xaeth> Regarding configs: For most systems and apps wouldnt the goal be to design templates so that there is minimal host specific configurations? 2008-07-31T16:53:42 <G> xaeth: thats how I'm going to lead the project yes 2008-07-31T16:53:45 <wakko666> mmcgrath: k... fp.o, koji, and fedorahosted, does that sound about right? 2008-07-31T16:53:51 <mmcgrath> yeah. 2008-07-31T16:53:53 <xaeth> G: kewl 2008-07-31T16:53:54 <mmcgrath> now here's the tricky part. 2008-07-31T16:53:58 <mmcgrath> G: where are you going to run it? 2008-07-31T16:54:00 <ricky> For web, I'd add fas as well 2008-07-31T16:54:07 <mmcgrath> hard to monitor fp.o from inside phx 2008-07-31T16:54:11 <G> oh well add fas1 too 2008-07-31T16:54:12 <ricky> And bodhi 2008-07-31T16:54:18 <wakko666> ricky: good idea. 2008-07-31T16:54:28 <G> okay, so hold on, make sure I've got my list right 2008-07-31T16:54:33 <ricky> G: app1-4 should be enough, as there are wsgi apps on those. 2008-07-31T16:54:57 <ricky> I mean the FAS/bodhi web interface, as probably goes down the most often ;-) 2008-07-31T16:55:03 <G> hosted1 hosted2 app1 app2 app3 app4 bastion nfs1 koji1 kojipkgs1 xenbuilder4 ppc4 and fas1 2008-07-31T16:55:23 <G> thats 13 2008-07-31T16:55:27 <G> oh actually... 2008-07-31T16:55:37 <G> forget fas1 add say xen7? 2008-07-31T16:56:12 <ricky> Interestingly, the xen servers are the simplest in terms of monitoring/services on them :-) 2008-07-31T16:56:17 <ricky> Could adding a db instead be useful? 2008-07-31T16:56:30 <mmcgrath> G: yeah, and for giggles lets also add the xen servers that each of those apps is on if they're on one. 2008-07-31T16:56:34 <mmcgrath> just so we can get the deps in there. 2008-07-31T16:56:40 <ricky> Oh yeah 2008-07-31T16:56:42 <G> mmcgrath: okay, I'll do that 2008-07-31T16:56:53 *** Standbye has quit IRC 2008-07-31T16:56:57 <xaeth> In my experience (and the manuals) u may see contention and slowness when running db and frontend together on 1 box w/ over 10 hosts 2008-07-31T16:57:01 *** Sparks has left #fedora-meeting 2008-07-31T16:57:02 <G> I think we'll cap it there so not to be monitoring the whole network for now 2008-07-31T16:57:20 <wakko666> G: do you mind if I take on configuring the web checks? 2008-07-31T16:57:22 <G> mmcgrath: for that, I think noc1 and db1 would be the best 2008-07-31T16:57:27 <xaeth> Will run, but u might start seeing gaps in data 2008-07-31T16:57:30 <G> wakko666: yeah 2008-07-31T16:57:56 <mmcgrath> G: I'll make that your call. If you want to go straght to db1 I'm fine with that. I did notice in the docs there's a db monitorer that will let us know when it goes down and zabbix is dead. 2008-07-31T16:58:08 <G> mmcgrath: yeah 2008-07-31T16:58:32 <mmcgrath> k 2008-07-31T16:58:35 <G> mmcgrath: as you said, db1 has plenty of power etc now so I think thats the right call 2008-07-31T16:58:59 <G> okay, D) What are we going to use as the metric of success? 2008-07-31T16:59:06 <mmcgrath> G: I agree, especially now that I know about the db monitorer. 2008-07-31T16:59:36 <mmcgrath> G: good question. I'd say a look at false positives. 2008-07-31T16:59:46 <G> sounds good to me 2008-07-31T16:59:56 <mmcgrath> as well as problems that we're monitoring that we didn't find out about. 2008-07-31T17:00:16 <G> false positives and false negatives 2008-07-31T17:00:31 <G> sounds good to moi 2008-07-31T17:00:42 <wakko666> notification latency might be another good metric. how long does it take for us to receive a notice of a problem. 2008-07-31T17:00:52 *** wfp has quit IRC 2008-07-31T17:01:12 <ricky> Based on the speed of the web interface updates, it's hopefully pretty good 2008-07-31T17:01:17 <G> wakko666: yeah, that'll be reasonable too 2008-07-31T17:01:46 <G> okay, it's past 9am here, so have we just about finished? 2008-07-31T17:02:27 <mmcgrath> G: yeah we should close the meeting now. 2008-07-31T17:02:34 *** hanthana has joined #fedora-meeting 2008-07-31T17:02:42 <G> mmcgrath: my thoughts exactly... 2008-07-31T17:02:44 <mmcgrath> anyone have anything else? we can continue the monitoring discussion in #fedora-admin 2008-07-31T17:02:47 * mmcgrath closes meeting in 30 2008-07-31T17:03:00 <fchiulli> If we find problems/want enhancements, do we know if upstream is receptive? 2008-07-31T17:03:07 <G> fchiulli: they are 2008-07-31T17:03:13 <G> and 1.6 looks pretty good for us too 2008-07-31T17:03:17 <mmcgrath> 10 2008-07-31T17:03:23 <mmcgrath> 5 2008-07-31T17:03:28 *** mmcgrath changes topic to "Infrastructure -- Meeting End" 2008-07-31T17:03:29 *** mcgarnicle has left #fedora-meeting 2008-07-31T17:03:32 <mmcgrath> thanks for coming everyone! 2008-07-31T17:03:38 <G> thank you! 2008-07-31T17:03:40 <SmootherFrOgZ> thanks 2008-07-31T17:03:48 *** mmcgrath changes topic to "Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel for meeting schedule"
_______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
