============================================ #fedora-meeting: Infrastructure (2014-04-24) ============================================ Meeting started by nirik at 18:00:48 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-04-24/infrastructure.2014-04-24-18.00.log.html . Meeting summary --------------- * aloha (nirik, 18:00:48) * New folks introductions and Apprentice tasks. (nirik, 18:03:09) * LINK: http://fedoraproject.org/easyfix/ (webpigeon, 18:14:47) * LINK: http://fedoraproject.org/easyfix/ (nirik, 18:14:47) * Applications status / discussion (nirik, 18:17:10) * new fedocal in prod this week (pingou, 18:17:28) * new pkgdb2 in stg this week (pingou, 18:17:37) * new copr release went out this morning with some minor bugfixes (nirik, 18:17:39) * Sysadmin status / discussion (nirik, 18:28:07) * sent out last announcement about ssl certs all redone for heartbleed (nirik, 18:28:49) * new virthost in ibiblio... ibiblio04 to replace 01 thats going end of life. (nirik, 18:29:17) * LINK: http://paste.fedoraproject.org/96714/64701139 is the full list (nirik, 18:38:50) * LINK: https://admin.fedoraproject.org/nagios/ (nirik, 18:44:02) * ideas to reduce nagios alerts welcome. (nirik, 18:45:07) * Upcoming Tasks/Items (nirik, 18:46:37) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:46:38) * LINK: https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 (threebean, 18:48:16) * Open Floor (nirik, 18:53:54) * LINK: https://github.com/stef/pywik (pingou, 18:56:21) Meeting ended at 18:59:57 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (108) * pingou (42) * threebean (18) * henderbj (17) * Penasio (14) * bwood09 (13) * adimania (11) * lmacken (7) * tflink (6) * ghostalker (6) * webpigeon (6) * kjs (6) * mirek-hm (6) * zodbot (5) * smooge (5) * relrod (3) * oddshocks (2) * janeznemanic (1) * blob (1) * mpduty (1) * dgilmore (1) * mdomsch (0) * puiterwijk (0) * abadger1999 (0) -- 18:00:48 <nirik> #startmeeting Infrastructure (2014-04-24) 18:00:48 <zodbot> Meeting started Thu Apr 24 18:00:48 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:48 <nirik> #meetingname infrastructure 18:00:48 <nirik> #topic aloha 18:00:48 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:48 <zodbot> The meeting name has been set to 'infrastructure' 18:00:48 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:01:11 <janeznemanic> hi 18:01:20 * adimania is here 18:01:24 * pingou 18:01:32 * ghostalker is here 18:01:54 <dgilmore> hola 18:02:23 * relrod here 18:02:23 * threebean is here 18:02:36 * bwood09 is here 18:03:06 <nirik> cool. lots of new folks too. ;) 18:03:09 <nirik> #topic New folks introductions and Apprentice tasks. 18:03:13 * lmacken 18:03:22 <nirik> anyone new like to give a short introduction of themselves? 18:03:28 <nirik> or apprentices with questions or comments? 18:03:31 <ghostalker> Sure 18:03:55 <ghostalker> I'm a 28 year old Sr Systems Engineer looking to take my operations experience and development skills to the next level, while giving back to the community. I'm also the one who didn't include the IRC nick in the email to the infra distro yesterday. 18:04:17 <nirik> welcome ghostalker! 18:04:25 <bwood09> Guess I should too 18:04:26 <pingou> welcome on board :) 18:04:29 <nirik> are you more interested in sysadmin type things? or application development? or both? 18:05:25 <ghostalker> A little of both. I have been just lurking since yesterday trying to get an idea of how things are structured. My development tends to be around helping do SysAdmin things. 18:05:49 <nirik> excellent. Always a need for sysadmin tooling too. :) 18:05:54 <nirik> bwood09: welcome and go ahead. ;) 18:05:55 <kjs> hi 18:05:58 <bwood09> I'm a (mostly hobbiest) sysadmin that is really into open source projects, but not a full-on programmer. Figure it'd be fun to try working with you guys 18:06:04 <ghostalker> As stated though my Python is weak so I would like to focus on SysAdmin related things will learning my way around. 18:06:13 <ghostalker> while* 18:06:32 <bwood09> I'm trying to work on my python skills too 18:06:41 <threebean> welcome :) 18:07:01 <Penasio> .fas 18:07:01 <zodbot> Penasio: (fas <query>) -- Search the Fedora Account System usernames, full names, and email addresses for a match. 18:07:07 <kjs> I'm interested in women, drugs and alcohol... 18:07:16 <henderbj> Hello all... first time on Infraestructure meeting 18:07:24 <Penasio> hi all... 18:07:28 <kjs> hi bj 18:07:32 <nirik> great. :) See me after the meeting over in #fedora-admin and I can get anyone interested added to our apprentice group: https://fedoraproject.org/wiki/Infrastructure_Apprentice 18:07:38 <Penasio> first time to me too 18:07:55 <pingou> it's cool to see so many new folks :) 18:08:17 <kjs> nirik: is kevin ? 18:08:36 <henderbj> Tnx, nirik... i want to get into apprentice group 18:08:40 <nirik> kjs: thats me, yes. well, one of them anyhow.... it's a somewhat common name. 18:08:56 <kjs> thought i recognised your nick 18:08:59 <blob> hello -- I'm interested in volunteering with releng and/or infrastructure. I've got quite a bit of sysadmin and development experience and currently work on a release engineering team for a large software project. I'm a long time fedora user and free software advocate that would like to give back. 18:09:19 <nirik> wow. lots of new folks ;) welcome blob as well. 18:09:39 <pingou> blob: cool 18:10:58 * oddshocks here late 18:11:03 * oddshocks in lecture 18:11:23 <pingou> oddshocks: enjoy 18:11:27 <nirik> Penasio / henderbj: care to introduce yourselves quickly and let us know if you are intested more in sysadmin or app devel? 18:11:51 <Penasio> My name is Claudio Penasio Jr. 18:11:54 * mpduty is here 18:12:13 <henderbj> Hello... i has been sysadmin a long time now... i can help there if you like 18:12:16 <Penasio> i'm Fedora user since 2003 18:12:37 <Penasio> sysadmin since 2000 18:12:45 <kjs> fyi im interested in sysadmin, but dont mind crossing the line every now and again, especially if it's security related. 18:12:49 <henderbj> I am willing to work in development too 18:12:57 <Penasio> i'm from ambassadors group 18:13:07 <webpigeon> O/ 18:13:29 <henderbj> I have experience programming... most on C, and Visual C, for Windows and a little for linux 18:13:39 <Penasio> 2009 to now i'm working on IT support manager 18:13:40 <nirik> excellent. :) Welcome everyone. I can add interested sysadmin folks to our apprentice group after the meeting over in #fedora-admin and we have #fedora-apps for our application development stuff... we can point you to easyfix tickets to look at getting started, etc. 18:13:43 <Penasio> :( 18:13:55 <henderbj> I programmed on Pascal, Matlab, PIC (assembler and C) 18:14:22 <Penasio> easyfix 18:14:25 <nirik> Most of our application development is python related and various python frameworks. ;) 18:14:36 <Penasio> look's like good 18:14:46 <henderbj> Ok... time to learn puthon then ;) 18:14:47 <webpigeon> http://fedoraproject.org/easyfix/ 18:14:47 <nirik> http://fedoraproject.org/easyfix/ 18:14:53 <nirik> :) 18:14:56 <webpigeon> :) 18:14:58 <Penasio> i'll need some menthoring 18:15:09 <henderbj> python, sorry!! 18:15:32 <threebean> lots of us can help ;) if you get stuck anywhere, be sure to ask (more than once.. sometimes, everyone is sleeping or something) 18:15:32 <nirik> Penasio: feel free to ask questions in any of our channels as you go and anyone available can answer and help you out. ;) 18:15:34 <henderbj> with u is kind of a bad word on spanish ;) 18:15:45 <nirik> henderbj: :) 18:16:06 <Penasio> thank's a lot 18:16:11 <pingou> henderbj: I guess it's close to the French one ^^ 18:16:16 <Penasio> i'm looking the easyfix 18:16:19 <nirik> ok, anyone else new who hasn't typed up yet? 18:16:58 <nirik> ok, great. ;) 18:17:02 <nirik> welcome again everyone! 18:17:10 <nirik> #topic Applications status / discussion 18:17:16 <henderbj> Tnx for warm welcome :) 18:17:21 <nirik> any application development news this week? 18:17:28 <pingou> #info new fedocal in prod this week 18:17:29 <nirik> I saw there was a new copr release... 18:17:37 <pingou> #info new pkgdb2 in stg this week 18:17:39 <nirik> #info new copr release went out this morning with some minor bugfixes 18:17:43 <pingou> (likely another one coming tomorrow) 18:17:48 <pingou> (pkgdb2 not copr :)) 18:18:01 <nirik> ok, cool. 18:18:14 <pingou> I spent today fix bugs and RFE on pkgdb2 after the announce I sent to the devel list 18:18:19 <pingou> which is nice :) 18:18:36 <lmacken> lots of bodhi2 development since PyCon. threebean has been working on a slick web fontend, and it's looking great. 18:18:41 * nirik looks for abompard. I wonder where we are with hyperkitty. Perhaps we could do some targeted testing on it and see about migrating a list or two? 18:18:43 <pingou> the other part of the day was spent on https://github.com/jmflinuxtx/kerneltest-harness/tree/master/frontend 18:19:02 <threebean> I bet abompard is vacationing still after pycon.. I think he was going to be travelling. 18:19:11 <pingou> which is a small app gather kernel test results 18:19:14 <nirik> for new folks: https://apps.fedoraproject.org/ has a overview of all these apps... where you can find more info on the source, etc. 18:19:27 <pingou> gathering* 18:19:37 <nirik> pingou: great. Thats redone in flask? or ? 18:19:43 <pingou> nirik: yes flask 18:19:54 <nirik> and it would use postgres? 18:20:00 <nirik> threebean: ah, gotcha 18:20:04 <pingou> Justing is working on import its data in it so we can test it a bit more 18:20:10 <pingou> I still have one idea for it 18:20:23 <pingou> nirik: yup, flask and sqlalchemy so we can use whatever we want :) 18:20:40 <threebean> great :) 18:20:43 <nirik> pingou: can you reply to the list post about that? so we keep people just watching the list in the loop 18:20:49 <pingou> nirik: will do 18:21:28 <threebean> i've been working on some monitoring stuff for fedmsg based on this stuff - http://threebean.org/blog/threading-moksha/ 18:21:37 <nirik> lmacken: is that up on the dev instance? or just local ahcking so far? 18:21:55 <threebean> i've got new moksha rpms in our staging environment that are working well now. soon to follow are some new nagios checks and collectd plugins. 18:22:29 <pingou> threebean: nice! 18:22:40 <pingou> threebean: you should blog about which approach you took finally :) 18:22:48 <threebean> oh, the last one. :) 18:23:09 <pingou> dedicated queue per thread 18:23:28 * threebean nods 18:23:36 <nirik> another thing I wanted to bring up: the bugzilla cookie/token change. :) Seems it's pushed back now, but we could still try and be ready. 18:23:37 <lmacken> nirik: just local right now, it's super easy to setup and import a db snapshot 18:23:39 <pingou> oh on the app front, I made some nice progress on gitsync https://github.com/pypingou/gitsync (also with the help of threebean :)) 18:24:11 * pingou sees the readme needs an update 18:24:15 <nirik> lmacken: nice. Question: is there anything private in bodhi db dumps? or could we make them public? 18:24:27 <lmacken> nirik: they've been public from day 1 of bodhi 18:24:44 <nirik> oh? I had no idea... :) where at? 18:24:54 <pingou> I also converted the DB model of mirrormanager to sqlalchemy, as a first step for MM2 :] 18:25:34 <threebean> nice. abadger1999 was talking about working on a mirrormanager rewrite. you may want to catch up with him about it when he's back from vacation. 18:25:39 <lmacken> nirik: https://fedorahosted.org/releases/b/o/bodhi/ - `make init` in bodhi1 pulls it in automatically, and the bodhi2 readme has instructions for importing it as well 18:25:50 <nirik> excellent. :) 18:25:52 <pingou> threebean: I've heard we would team up ;-) 18:26:46 <nirik> ok, any other application news or questions or comments? 18:27:24 <smooge> the sky is blue 18:27:30 <pingou> smooge: grey here 18:27:41 <adimania> dark black here :( 18:27:45 <nirik> sort of a off white ceiling color here. ;) 18:28:07 <nirik> #topic Sysadmin status / discussion 18:28:34 <nirik> ok, on the sysadmin side of things... 18:28:49 <nirik> #info sent out last announcement about ssl certs all redone for heartbleed 18:28:58 <nirik> hopefully we are done with heartbleed stuff now 18:29:17 <nirik> #info new virthost in ibiblio... ibiblio04 to replace 01 thats going end of life. 18:29:49 <bwood09> I have a quick question 18:29:50 <nirik> new machines in phx2 are going to be installed by smooge over the next few weeks. 18:29:56 <nirik> bwood09: sure, fire away anytime. 18:30:34 <bwood09> I was looking at the easyfix list; why are we switching from puppet to ansible? Since I'm new, I'd like to hear the thought behind that. 18:30:50 <bwood09> (Ticket 4220) 18:30:56 <nirik> bwood09: I suppose I should make a wiki page for it or something, as people keep asking about it. ;) 18:31:08 <nirik> basically ansible meets our needs better. 18:31:42 * mirek-hm is late 18:32:00 <relrod> I started working on porting the puppet nagios server module to ansible this morning, but still have a long way to go. No ETA yet. 18:32:37 <nirik> It's simpler, it's in python (and we are mostly a python shop), it's agentless so we don't need to run a agent on all machines, it uses ssh instead of a seperate protocol, it allows us to manage things that are different oses more easily (fedora and rhel), etc 18:32:45 <bwood09> alright 18:32:50 <henderbj> sysadmin work has levels, like level 1, level 2, level 3? 18:32:50 <bwood09> Makes sense, actually 18:33:11 <nirik> relrod: cool. threebean had some ideas around that a while back I think... how to autogenerate much of it? 18:33:54 <nirik> henderbj: well, there's various groups... the apprentice group allows people to login to a bunch of machines and see our git repos, etc... but it's read only, you need someone else to review and make the actual change... 18:33:55 <henderbj> where i worked, level 1 are those facing clients, level 2 is medium problems, and level 3 for developers problems 18:33:59 <relrod> hm ok. I just started doing a straight port, I figured we could automate it later if we decide to...but if we want to do it as we port, we can try 18:34:48 <nirik> henderbj: next are sysadmin-something groups... those are areas that can make changes... like 'sysadmin-noc' is people who can login to machines, has sudo on some and can commit changes, etc. 18:34:58 <threebean> relrod: yeah, I didn't have much :) 18:35:14 <henderbj> got it... tnx 18:35:18 <nirik> then there is sysadmin-main, which is the folks who have access to everything... thats a small group that people are added to only after a long record of contribution. ;) 18:35:35 <threebean> relrod: we could probably automatically determine things like dependencies between hosts and guests. but services get trickier. 18:35:37 <nirik> so, speaking of nagios... 18:35:38 <henderbj> i read about it a little before 18:35:47 <pingou> nirik: and few craig of beers 18:36:03 <nirik> I wanted to do a new thing in the meeting every week in this section... 18:36:22 <nirik> I want to go over the last weeks nagios alerts, and see if we can come up with ways to cut them down. ;) 18:36:23 <pingou> oh, shiny new things every week! :) 18:36:33 <adimania> +1 18:36:48 <adimania> I am doing something every similar these days at work. 18:37:08 <adimania> We generate a report of all the alert sent out every day and review it. 18:37:27 <nirik> adimania: cool. nagios? do you have a script for it that we could reuse? 18:37:34 <nirik> I just manually looked before the meeting. ;) 18:37:49 <nirik> there were 77 alerts in the last 7 days. 18:38:00 <nirik> 11 of those were from unbound-telia01 18:38:14 <adimania> We just send a metric to graphite every time we create any noise. 18:38:31 <nirik> 29 of them were from network problems at coloamerica... various instances there 18:38:31 <adimania> and do an aggregate at the end of the day. 18:38:50 <nirik> http://paste.fedoraproject.org/96714/64701139 is the full list 18:40:05 <nirik> so, I think I will adjust unbound-telia01 to do more retries before alerting... or longer timeframe. 18:40:32 <nirik> any other patterns folks can see or things we could adjust? 18:41:13 <threebean> at very least, it can give admin apprentices something to look for in the coming week 18:41:17 <nirik> if you come up with any, please do let me know or propose changes. ;) 18:41:21 <pingou> ftr, there are a *bunch* of email on the queue on bastion 18:41:35 <pingou> -- 77340 Kbytes in 7426 Requests. 18:41:37 <nirik> pingou: thats always been the case... it's the mail gateway... so it gets deferred stuff 18:41:42 <henderbj> disconnection alerts are good 18:41:43 <pingou> ok 18:42:03 <henderbj> they inform about poor connections, bottlenecks, or something 18:42:22 <nirik> another source of alerts for a while (or perhaps it was just emails) was fas02... it was getting overloaded from time to time. 18:42:34 <webpigeon> couple of disk space alerts in there 18:42:34 <adimania> swap alerts? 18:42:35 <nirik> I tweaked it's config some and it's been much better 18:42:39 <henderbj> maybe the other side has high cpu usage, etc 18:43:41 <nirik> yeah, a few swap alerts... usually thats when something gets hammered with requests or there's some problem processing. 18:43:55 <nirik> you can all also look at nagios directly to see the exact times, etc 18:44:02 <nirik> https://admin.fedoraproject.org/nagios/ 18:44:40 <nirik> also, there's a few easyfix tickets I filed about adding some machines to nagios. ;) 18:45:07 <nirik> #info ideas to reduce nagios alerts welcome. 18:45:20 <nirik> Finally on the sysadmin side, there may be a outage next week... 18:45:24 <adimania> I can pick that up, if no one else is looking for easyfixes. 18:45:41 <nirik> the storage folks are finally hoping to move our backend storage. They say it won't cause any problems, but we should schedule an outage just in case. 18:45:54 <pingou> +1 18:46:03 <nirik> adimania: feel free. I need to file another one too.... lots for everyone. ;) 18:46:22 <nirik> any other sysadmin news/questions/comments? 18:46:37 <nirik> #topic Upcoming Tasks/Items 18:46:38 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:46:46 <nirik> anything upcoming folks would like to note or schedule? 18:46:47 * webpigeon still needs to refine his dns script, but hasn't had time with work 18:46:50 <nirik> threebean: hows the fad looking? 18:47:01 <adimania> FUDCon APAC is in less than a month. I am looking for a really really easyfix puppet module which I can migrate to Ansible in a probable workshop. 18:47:06 <nirik> webpigeon: no worries. we will get there when we do. ;) 18:48:08 <threebean> nirik: good, I think. we've got hotels and flights set at this point. 18:48:16 <threebean> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 18:48:17 <pingou> threebean: nice :) 18:48:51 <nirik> cool. 18:48:57 <mirek-hm> I am working on ansible manifest to install recent open stack - I am sowhere in half a way. If somebody want to help, just ping me 18:49:02 <nirik> I still need to send out info asking for interest in an ansible one. 18:49:07 <threebean> unless I'm forgetting something, we just need to get there and make the magic happen. 18:49:30 <nirik> threebean: excellent. ;) 18:49:48 <nirik> mirek-hm: thanks for working on that... are you basing on icehouse? 18:49:58 <mirek-hm> nirik: yes 18:50:02 <nirik> great 18:50:56 <tflink> threebean: are there any plans to work on the 1st? I'm trying to figure out when to head up to Denver 18:50:57 <nirik> oh, one other thing upcoming... not next week, but the week after, I will be out mon-wed... really the 1st-5th. 18:51:06 <tflink> mirek-hm: what kind of help are you looking for? 18:51:47 <mirek-hm> tflink: at least some review would be welcome 18:51:52 <threebean> tflink: I was hoping to work then, but we haven't sorted that out. bocecha is actually getting in on the 31st. 18:51:57 <pingou> nirik: the same week, I'll be out, Mond to Friday, 5th to 9th 18:52:27 <mirek-hm> tflink: or take some chapter from installation guide and implement it (that would be super cool) 18:52:39 <bwood09> Oh, wow, Nagios let me log in with my creds 18:53:02 <nirik> bwood09: yeah, should be open to any fas accounts for read only. It likely won't let you ack alerts or anything tho 18:53:10 <bwood09> Figured as much 18:53:44 <nirik> but you can see history and such. 18:53:54 <nirik> #topic Open Floor 18:53:59 <nirik> anyone have items for open floor? 18:54:18 <nirik> questions, comments, favorite chocolate? 18:54:22 <bwood09> nirik, also, I mentioned in my welcome email that I'm unemployed, I am planning on devoting my time to this pretty much like a full time job. That's not a bad thing, is it? lol 18:54:24 <smooge> blue 18:54:27 <pingou> nirik: do we have any web-monitor solution in place? 18:54:38 <adimania> how about the slimming advice :P 18:54:43 <pingou> I remember we had awstats but it's not working so much it is? 18:54:53 <nirik> bwood09: :) you're welcome to spend as much time as you like... just don't get burned out. :) 18:55:06 <tflink> threebean: I was waiting on that before making a hotel reservation but I suppose that I could just commute for a day or two. it's only an hour or so 18:55:09 <nirik> pingou: for web logs? or monitoring web pages for alerts? 18:55:17 <bwood09> I'll try not to. I really, really like doing sysadmin work 18:55:17 * pingou wonders if there is an equivalent of piwik in python 18:55:20 <smooge> fedoramagazine is updated 18:55:28 <pingou> nirik: web-logs 18:55:29 <nirik> smooge: cool! :) 18:55:36 <tflink> mirek-hm: where are the playbooks that you're working on? 18:55:36 <smooge> pingou, look for pywik? 18:55:38 <webpigeon> nirik: Galaxy, as it happens. (but cool that nagios lets people login with their FAS creds) 18:55:42 <nirik> pingou: we don't, but it would be good if we did. 18:55:45 <pingou> smooge: does it? :D 18:56:19 <smooge> pingou, maybe pywyk? or pypiwik? [I am just pulling your leg now.] 18:56:21 <pingou> https://github.com/stef/pywik 18:56:34 <mirek-hm> tflink: on my machine, only. But I will push them to github and send link to infra mailing list so you can review if you want, ok? 18:56:53 <nirik> mirek-hm: that would be great. 18:56:59 <threebean> tflink: well, we're all only in town for so long and I know we've got tons of stuff to do.. so I'd vote for "yes, let's plan to hack on June 1st" 18:57:28 <tflink> mirek-hm: yeah, that'd be great 18:57:48 <nirik> threebean: +1 18:57:54 <tflink> threebean: ok, I'll double check to make sure that mike can do the 1st as well 18:58:01 <lmacken> yeah, I'm not sure when people are showing up, but I'm cool with hacking any time :) 18:58:53 * nirik too 18:59:11 <lmacken> let's talk fad logistics in #fedora-apps. we need to figure out what to do other than slay code :) 18:59:15 <nirik> also, we should definitely be active in our usual channels and #fedora-fad during the fad. :) 18:59:49 <nirik> alright, we are out of time. Lets continue over in #fedora-admin, #fedora-apps and #fedora-noc. :) 18:59:53 <nirik> thanks for coming everyone! 18:59:57 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
