============================================ #fedora-meeting: Infrastructure (2014-10-02) ============================================ Meeting started by nirik at 18:00:00 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-10-02/infrastructure.2014-10-02-18.00.log.html . Meeting summary --------------- * aloha (nirik, 18:00:01) * New folks introductions and Apprentice tasks. (nirik, 18:02:01) * Applications status / discussion (nirik, 18:03:38) * LINK: https://github.com/fedora-infra/pkgdb2/blob/master/utility/pkgdb2.spec#L114 (pingou, 18:04:12) * pkgdb2 update in prod today, another minor one tomorrow. (nirik, 18:07:18) * anitya almost ready for production. needs dns to finish getting setup. (nirik, 18:07:52) * taskotron almost ready for production. (nirik, 18:08:03) * hyperkitty is working on sqlalchemy changes. Will deploy after those land. (nirik, 18:15:47) * bodhi2 is in jenkins now and running tests on commits (nirik, 18:16:19) * ACTION: pingou to coordinate with tyll on cnucnu bugzilla filing under the new anitya setup. (nirik, 18:24:40) * fedora mobile getting setup to be updated from the google play alpha track. (nirik, 18:25:05) * Sysadmin status / discussion (nirik, 18:26:27) * mass reboot earlier this week. Most things are rebooted/updated (nirik, 18:26:59) * kernel01/02 now have a bunch more memory. Thanks smooge for getting that installed. (nirik, 18:30:15) * db04 (rhel6) migrated to db-koji01 (rhel7) (nirik, 18:30:33) * 0 guests left at telia01 (nirik, 18:30:43) * 78 hosts left in puppet (nirik, 18:33:28) * nagios/alerts recap (nirik, 18:33:44) * Upcoming Tasks/Items (nirik, 18:40:02) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:40:03) * Open Floor (nirik, 18:41:58) Meeting ended at 18:47:43 UTC. Action Items ------------ * pingou to coordinate with tyll on cnucnu bugzilla filing under the new anitya setup. Action Items, by person ----------------------- * pingou * pingou to coordinate with tyll on cnucnu bugzilla filing under the new anitya setup. * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (109) * pingou (36) * relrod (15) * smooge (11) * abompard (6) * lmacken (6) * oddshocks (6) * tflink (6) * zodbot (5) * threebean (2) * RogerBTX (1) * Neldogz (1) * danielbruno (1) * lanica (1) * mpduty (1) * dgilmore (1) * puiterwijk (0) * abadger1999 (0) * mdomsch (0) -- 18:00:00 <nirik> #startmeeting Infrastructure (2014-10-02) 18:00:00 <zodbot> Meeting started Thu Oct 2 18:00:00 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 <nirik> #meetingname infrastructure 18:00:01 <nirik> #topic aloha 18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:01 <zodbot> The meeting name has been set to 'infrastructure' 18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:14 * threebean is here 18:00:14 * pingou 18:00:18 * tflink is here 18:00:19 * relrod here 18:00:26 * lmacken 18:00:28 * danielbruno is here 18:00:46 * oddshocks here 18:01:22 * RogerBTX present @ laundromat 18:01:33 * mpduty is here 18:01:58 <Neldogz> neldogz is here 18:02:01 <nirik> #topic New folks introductions and Apprentice tasks. 18:02:13 <nirik> any new folks like to introduce themselves today? 18:02:20 <nirik> or apprentices with questions, comments or ideas? 18:03:24 <nirik> ok. ;) moving along then... 18:03:38 <nirik> #topic Applications status / discussion 18:03:45 <nirik> any application news this week? :) 18:03:52 <pingou> new pkgdb in prod today 18:04:04 <nirik> nice. any list of changes? 18:04:11 <smooge> mgoodmday 18:04:12 <pingou> https://github.com/fedora-infra/pkgdb2/blob/master/utility/pkgdb2.spec#L114 18:04:23 * nirik wonders if we shouldn't try and mail at least infra list on updates? just a short 'pkgdb2 updated, here's changes' 18:04:41 <pingou> I added the update_package_info script that was written a while ago 18:05:01 <pingou> whose goal is to update the package's info (summary and description) from yum's info (thus the rpm) 18:05:20 <pingou> also, now the user orphaning a package will loose his/her commit and approveacl ACL 18:05:34 <nirik> ok. I guess if they need it back they can unorphan. ;) 18:05:40 <pingou> yes :) 18:05:48 <pingou> the rest is mostly cosmetic changes in the UI 18:06:07 <pingou> there will be another release tomorrow to fix something in the emails pkgdb send when it has a problem 18:06:16 <pingou> and I have a new fedocal release coming after that 18:06:23 <nirik> busy busy. ;) 18:06:32 <pingou> (w/ for example the possibility to send a reminder to multiple email addresses) 18:06:40 <tflink> taskotron production is making progress 18:06:52 <pingou> and anitya is ready in prod, just waiting for the DNS to sync in: http://140.211.169.229/ 18:07:06 <pingou> but I'll call for tester before calling it 1.0 :) 18:07:13 <tflink> waiting on two bits of code, learning the hard way about bits that need to be monitored after restart etc. 18:07:18 <nirik> #info pkgdb2 update in prod today, another minor one tomorrow. 18:07:45 * dgilmore is kinda here but really not 18:07:52 <nirik> #info anitya almost ready for production. needs dns to finish getting setup. 18:08:03 <nirik> #info taskotron almost ready for production. 18:08:28 <nirik> tflink: aside from those issues everything looking ok from a infra standpoint for you? I know we need monitoring still. 18:08:54 <tflink> looking into some vpn issues ATM 18:09:04 * pingou needs to add a nagios check for anitya 18:09:10 <tflink> monitoring, backup (ticket hasn't been filed yet) 18:09:28 <nirik> ah yeah I am already backing up the db... 18:09:34 <nirik> but there's possibly more. 18:09:51 <tflink> there are some files on taskotron01.qa that will need to be backed up 18:10:00 <nirik> ok. 18:10:07 * lanica is here for the infra meeting (sorry I'm late!) 18:10:14 <nirik> welcome lanica 18:10:57 <nirik> ok, anything else pending on the applications side? 18:11:29 <threebean> eh, I've been doing lots of monitoring stuff this week and will be pushing out some other changes for anitya later. 18:11:41 <nirik> cool. 18:12:18 <oddshocks> general improvements to fedimg and ostree/atomic stuff. nothing outstanding. 18:12:23 <oddshocks> making progress 18:12:37 <pingou> cool 18:12:46 <nirik> abompard: whats the current word on hyperkitty (if you happen to be around). I'd be fine moving some lists if there's nothing we are waiting on. 18:12:52 <nirik> oddshocks: cool. 18:13:19 <abompard> nirik: actually there's currently lots going on around Mailman's SQLAlchemy port 18:13:26 <abompard> it may be happenning soon 18:13:42 <lmacken> I got bodhi2 test suite back up and running on jenkins this week. Lots of masher hacking as well, but that hasn't hit infra yet. http://jenkins.cloud.fedoraproject.org/job/bodhi/ 18:13:54 <abompard> so I'm focusing on that, the windows of availability for barry to review code are small 18:13:56 <nirik> abompard: nice. :) do we want to wait for that? 18:14:15 <abompard> nirik: yes, if it happens as soon as I think it can 18:14:25 <nirik> lmacken: nice. 18:14:25 <oddshocks> abompard: as in warsaw? 18:14:30 <abompard> oddshocks: yep 18:14:33 <oddshocks> abompard: niceee 18:14:48 <nirik> lmacken: no weird disappearances on jenkins lately? did we ever figure out what was happening with that? 18:15:03 <pingou> abompard: champagne \ó/ 18:15:14 <abompard> pingou: hell yeah 18:15:33 <nirik> cool. 18:15:38 <lmacken> nirik: as far as plugins disappearing, I think the ansible playbook overwrites them all, and we don't have the fedmsg plugin in git 18:15:47 <nirik> #info hyperkitty is working on sqlalchemy changes. Will deploy after those land. 18:15:54 <nirik> lmacken: ah, that would do it. 18:15:56 <lmacken> as for projects disappearing, I have no idea how that happened to begin with, but it has yet to happen again 18:16:19 <nirik> #info bodhi2 is in jenkins now and running tests on commits 18:16:21 <oddshocks> jenkins is just trying to get out of work. slacker. 18:16:22 <pingou> yeah the plugins need to be installed via the ansible playbook 18:16:52 <pingou> not via jenkins' UI (or done in both place, the playbook in addition to the UI) 18:17:30 <nirik> ok. I think relrod setup the fedmsg plugin there? relrod and pingou: can you see if you can get that in ansible (if it's not) 18:17:56 <relrod> yeah, we need to ansibleize it 18:18:03 <nirik> cool 18:18:09 <pingou> relrod: there is also a problem with the yum repo that was added 18:18:11 <pingou> (the copr one) 18:18:33 <relrod> pingou: yeah, I know. Jenkins can't talk to copr for some reason. Short term solution is put the RPM from that copr in the infra repo 18:18:36 <nirik> yeah, we talked about that... 18:18:37 <relrod> I just need to actually do it 18:18:44 * nirik nods. 18:18:49 <pingou> relrod: ok cool 18:19:16 <nirik> ok, anything else on the applications side? 18:19:43 <nirik> pingou: oh, side question: does anitya file bugzilla bugs on things? or was that a different layer of our old cnucnu thing? 18:19:44 <relrod> one fedora mobile related thing I guess: 18:20:01 <pingou> nirik: it'll be a different layer 18:20:23 <nirik> pingou: ok, was that something tyll was running? does he know to update? or is that something we want to run? 18:20:30 <relrod> I am working on a system for pushing out nightly updates to it via the Google Play alpha track. This means we can lose the ugly self-updating code that is in Mobile right now. 18:20:49 <nirik> relrod: cool. 18:20:51 <relrod> We were going to do this via Jenkins (https://github.com/fedora-infra/mobile/issues/41) but ran into some issues 18:21:02 <pingou> nirik: we'll probably want a fedmsg-cnucnu as we have fedmsg-fasclient 18:21:09 <relrod> so it's probably going to be a separate process that just pulls the latest APK from Jenkins and sends it to Google 18:21:12 <relrod> which is fine 18:21:20 <pingou> nirik: I'll coordinate w/ tyll 18:21:49 <nirik> relrod: ok 18:21:53 <nirik> pingou: sounds good 18:22:02 <relrod> (the issue is basically "the APK has to be signed, and we don't really want the signing key for it on Jenkins") 18:22:52 <nirik> relrod: hum, ok... where would that live then/ 18:22:53 <nirik> ? 18:23:36 <nirik> or TBD? 18:23:40 <relrod> nirik: Probably ansible private repo, and the nightly pusher can be a cloud node or something? Doesn't need to be very powerful 18:23:47 <nirik> ok 18:24:23 <relrod> If we don't want it in the ansible private repo, open to better suggestions 18:24:40 <nirik> #action pingou to coordinate with tyll on cnucnu bugzilla filing under the new anitya setup. 18:24:47 <nirik> I think that would be ok 18:25:05 <nirik> #info fedora mobile getting setup to be updated from the google play alpha track. 18:25:11 <relrod> I'm not sure if the alpha signing key can be different than the production track signing key (if anyone has Android experience and knows, poke me? :)) 18:25:27 <nirik> no idea off hand. 18:25:39 * nirik waits for the ffos html 5 version. ;) 18:25:48 <relrod> :P 18:25:56 <relrod> anyway, that's all I have. 18:26:00 <nirik> cool. thanks. 18:26:12 <nirik> anything else application wise? or shall I move on? 18:26:27 <nirik> #topic Sysadmin status / discussion 18:26:35 <nirik> so, we did a mass reboot earlier this week. 18:26:47 <nirik> there's a few stragglers we still need to do, but mostly everything is done. 18:26:59 <nirik> #info mass reboot earlier this week. Most things are rebooted/updated 18:27:18 <nirik> I migrated db04 (rhel6) to db-koji01 (rhel7) 18:27:47 <nirik> I killed db04 and keys01. We now have 0 guests at telia01 18:28:04 <smooge> <<plays taps>> 18:28:10 <pingou> I need to check what specific settings we are putting on the postgresql.conf file we install with the postgresql_server role, as this seems to be closely related to the memory of the host 18:28:20 <nirik> I'm also going to retire mirrorlist-serverbeach, because it keeps having problems keeping up. 18:28:26 <pingou> by telia01 18:28:31 <pingou> bye* 18:28:45 <nirik> pingou: yeah, I think we should be more dynamic on it. 18:29:03 <nirik> pingou: also, I think we just copied our old postgresql.conf in... we should get the rhel7 one and adjust it. 18:29:09 <pingou> make it a template and use host_vars or so 18:29:12 <nirik> because we might be missing other things the 7 one can do better 18:29:18 <pingou> also true 18:29:27 <nirik> if you want to work on that that would be great. ;) 18:29:33 <pingou> anitya-backend should have the default el7 one 18:29:41 <pingou> I can at least provide a diff :) 18:29:59 <nirik> ok 18:30:15 <nirik> #info kernel01/02 now have a bunch more memory. Thanks smooge for getting that installed. 18:30:33 <nirik> #info db04 (rhel6) migrated to db-koji01 (rhel7) 18:30:39 <smooge> well thanks jesse. all I did was watch from afar :) 18:30:43 <nirik> #info 0 guests left at telia01 18:30:58 <pingou> nirik: how many guests left in puppet? 18:31:35 <nirik> 78 18:31:45 <pingou> k 18:31:48 <nirik> I'm going to try and move some more next week... 18:32:06 <nirik> a number are virthosts, so we will have to move all the guests, update, move back 18:32:23 <pingou> that'll be fun 18:32:34 <nirik> and of course we still have proxies to convert. thats 8 there 18:33:03 <nirik> but I am going to try and get there before the end of the year if we can. 18:33:28 <nirik> #info 78 hosts left in puppet 18:33:44 <nirik> #topic nagios/alerts recap 18:33:56 * nirik digs up url. where's puiterwijk with it handy when you need him. ;) 18:35:21 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=10&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=10&eday=2&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25 18:35:23 <zodbot> nirik: http://tinyurl.com/pcpaya7 18:35:39 <nirik> download01.mgmt was fixed, was a IMM going wonky 18:36:08 <nirik> the datagrepper I think was the datagrepper db migration that threebean did 18:36:27 <nirik> the collab03 mail queue is due to the way that check is written. We should fix it. 18:36:47 <nirik> it looks for anything mroe than 3 or 4 emails in queue, but thats our list server, so sometimes it has a bunch in there it's sending out. 18:36:57 <smooge> I rebuilt bvirthost10 on the UCS cisco. much swearing and pain 18:37:02 <smooge> oops sorry 18:37:05 <nirik> thanks for that smooge 18:37:26 <nirik> mirrorlist-serverbeach is going away, so that should disappear. 18:38:01 <smooge> <<taps on lone bugle>> 18:38:15 <nirik> not sure about the bodhi02 ones... lmacken: any errors you have seen from it lately? 18:38:26 <nirik> might have been the koji outage for the database move? 18:39:29 <nirik> anything else sysadminy? 18:40:02 <nirik> #topic Upcoming Tasks/Items 18:40:03 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:40:11 <nirik> anything upcoming anyone would like to note or schedule? 18:40:33 <nirik> I'll note we go into Beta freeze 2014-10-14 18:40:45 <nirik> so do try and get anything done before then thats big/disruptive. 18:41:05 <pingou> 2 weeks from now 18:41:08 <nirik> yep. 18:41:09 <pingou> that'll be there soon 18:41:23 <nirik> I'm going to try and get some more rhel7/ansible migrations done, but we will see 18:41:42 <nirik> might do an outage and move db01 over. 18:41:47 <nirik> will have to look. 18:41:58 <nirik> #topic Open Floor 18:42:15 <nirik> anyone have items for open floor? ideas, comments, cookie recipes? 18:42:17 <smooge> I have two cloud servers to rebuild today. 18:42:31 <smooge> and I will be out tomorrow after the EPEL meeting 18:42:55 <nirik> smooge: cool. Let me know when 09 is done and I can play with running the setup on it. 18:43:31 <smooge> will do so 18:43:47 <smooge> next week I will be focusing on I2 and budgets 18:43:48 <nirik> If we can get that all working soon we can look at cloud migrating. ;) 18:43:56 <smooge> or cloud 18:44:09 <lmacken> nirik: no I haven't seen any errors from bodhi02 aside from the koji outage. 18:44:31 <nirik> I think thats mostly just a matter of telling everyone: you have until xxxx-xx-xx to terminate your instance(s) and bring them up in the new cloud. 18:44:43 <nirik> and us doing that on the persistent ones we care about. 18:44:54 <nirik> oh, we still need to get storage working tho. 18:44:58 <nirik> lmacken: ok 18:45:09 <nirik> lmacken: just odd that bodhi02 was in the nagios alerts, but not 01... 18:45:32 <lmacken> nirik: weird, I'll take a look 18:46:11 <nirik> thanks. 18:46:18 <nirik> ok, if nothing else, will close out in a minute or so... 18:47:41 <nirik> Thanks for coming everyone! Lets all continue in #fedora-admin, #fedora-apps and #fedora-noc. 18:47:43 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
