============================================ #fedora-meeting: Infrastructure (2013-04-11) ============================================ Meeting started by nirik at 19:00:01 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2013-04-11/infrastructure.2013-04-11-19.00.log.html . Meeting summary --------------- * welcome y'all (nirik, 19:00:01) * New folks introductions and Apprentice tasks. (nirik, 19:01:36) * Applications status / discussion (nirik, 19:02:41) * threebean fixed email notifications in askbot. hurray! (nirik, 19:03:11) * Sysadmin status / discussion (nirik, 19:08:04) * hopefully will see new netapp space at some point for /mnt/koji and hopefully backups (nirik, 19:10:07) * we have all our arm boxes installed, but are going to look at testing replacing disk on one or two with ssd to see how much that helps builds. (nirik, 19:10:30) * some ansible poking ongoing (nirik, 19:10:47) * Private Cloud status update / discussion (nirik, 19:21:12) * Upcoming Tasks/Items (nirik, 19:25:52) * 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze (nirik, 19:26:03) * 2013-04-15 - remove legacy-openid (nirik, 19:26:03) * 2013-04-15 - announce openid change on hosted. (nirik, 19:26:03) * 2013-04-15 - drop inactive apprentices (nirik, 19:26:03) * 2013-04-16 F19 alpha release (nirik, 19:26:03) * 2013-04-17 - switch to new sigul servers. (nirik, 19:26:04) * 2013-04-19 - switch hosted to openid (nirik, 19:26:05) * Open Floor (nirik, 19:27:23) * LINK: https://fedoraproject.org/wiki/Infrastructure_FedoraHosted_Version2_Notes (nirik, 19:28:55) Meeting ended at 19:31:54 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (89) * skvidal (52) * threebean (14) * smooge (9) * pingou (6) * abadger1999 (5) * zodbot (4) * lmacken (2) * oddshocks (2) * frankly3d (2) * samkottler (1) * biker (1) * jerzyr (1) * relrod (1) * ricky (0) * mdomsch (0) * dgilmore (0) * CodeBlock (0) -- 19:00:01 <nirik> #startmeeting Infrastructure (2013-04-11) 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <zodbot> Meeting started Thu Apr 11 19:00:01 2013 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:01 <nirik> #topic welcome y'all 19:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 19:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 19:00:11 <nirik> who all is around for a nice infra meeting? 19:00:13 * skvidal is here but grumbly about food 19:00:13 * samkottler is here 19:00:15 * oddshocks is here 19:00:19 <jerzyr> here 19:00:23 * lmacken here 19:00:27 * pingou here 19:00:31 <frankly3d> here 19:00:46 * threebean here 19:01:01 * abadger1999 here 19:01:30 <nirik> cool. 19:01:34 <nirik> lets go ahead and dive in 19:01:36 <nirik> #topic New folks introductions and Apprentice tasks. 19:01:43 <nirik> any new folks? or apprentices with questions? 19:01:53 <oddshocks> things are going well on my end 19:01:54 <smooge> here 19:02:03 <skvidal> smooge: you're not new! 19:02:18 <skvidal> smooge: :-D 19:02:25 <nirik> oddshocks: glad to hear. :) 19:02:41 <nirik> #topic Applications status / discussion 19:02:50 <nirik> any apps news? note that we are still in freeze. 19:02:58 * pingou mainly bothered threebean 19:03:09 * threebean bothered pingou 19:03:11 <nirik> #info threebean fixed email notifications in askbot. hurray! 19:03:16 * biker here 19:03:21 <pingou> threebean: I bothered you more :-p 19:03:30 <threebean> nuh-uh! :P 19:03:47 <frankly3d> Behave Children! 19:03:51 <threebean> on the askbot front, it's an issue with python-lxml on 32bit rhel6 boxes. 19:04:01 <nirik> we have fedocal and blockerbugs in stg... so possibly they will go to prod after this alpha freeze is over. 19:04:09 <nirik> threebean: really? crazy 19:04:29 <threebean> I filed an bug about it against the rhel package, but we should reinstall ask01 as 64bit as soon as is convenient. 19:04:47 <pingou> abadger1999: btw, would you mind to test a bit fedocal, I'm not sure how it will behave with non-ascii chars 19:04:57 <nirik> threebean: so, we are just working around it now? 19:05:01 * threebean nods 19:05:17 <threebean> I patched askbot to not use lxml. 19:05:17 <nirik> threebean: I think we did 32bit due to the python memory issues... 19:05:18 <abadger1999> Just a note that I'm behind on the pkgdb release -- dove into a few Fedora packaging messes and porting python-fedora away from paver instead. 19:05:26 <nirik> but I guess we can try 64 and see how it is. 19:05:42 <threebean> nirik: interesting. I didn't know that. ask01.stg is 64bit which made things a little confusing. 19:06:00 * relrod is here, a bit late. 19:06:06 <abadger1999> pingou: https://fedocal.dev.fedoraproject.org/ ? I can browse aroundand see if I get 500s :-) 19:06:12 * nirik is confused then, which box is 32bit? 19:06:18 <pingou> abadger1999: that one or the .stg. one 19:06:24 <nirik> oh, prod vs stg. got it. 19:06:37 <abadger1999> pingou: What's the stg url? 19:06:45 <threebean> :) yeah :) 19:06:54 <pingou> abadger1999: admin.stg.fedoraproject.org/calendar 19:06:54 * nirik adds 'reinstall ask01 as 64bit' to list. 19:07:04 <abadger1999> pingou: Cool. I'll browse around there. 19:07:49 <nirik> ok, any other applications news? 19:08:04 <nirik> #topic Sysadmin status / discussion 19:08:04 <threebean> Oh, there's talk of debian picking up fedmsg -> http://lists.debian.org/debian-qa/2013/04/msg00010.html 19:08:11 <nirik> threebean: oh yeah? wow. 19:08:40 <threebean> yeah, nothing super concrete yet, but olasd in #fedora-apps is looking to mentor someone on it. 19:08:42 <nirik> awesome. 19:08:46 <threebean> I'll be keeping in touch with them :) 19:08:58 <nirik> I wonder if we could setup some kind of meta fedmsg of all the fedmsg using places... 19:09:07 * threebean nods 19:09:14 <nirik> cool. 19:09:15 <threebean> it would be trivial to subscribe to both streams 19:09:22 <nirik> so, on sysadmin side of things... 19:10:07 <nirik> #info hopefully will see new netapp space at some point for /mnt/koji and hopefully backups 19:10:30 <nirik> #info we have all our arm boxes installed, but are going to look at testing replacing disk on one or two with ssd to see how much that helps builds. 19:10:47 <nirik> #info some ansible poking ongoing 19:10:54 <skvidal> so much poking 19:11:02 <nirik> I whipped up a fas-client ansible module, but I haven't tested it yet. 19:11:12 <nirik> but hopefully soon. 19:11:20 <skvidal> and the new roles mechanism in ansible 1.2/git is going to make us restructure a bunch of things 19:11:21 <nirik> skvidal: whats the status on the cron/trigger stuff? 19:11:22 <skvidal> for beauty 19:11:40 <lmacken> AnsibleFest 2013 in Boston Jun 12th 19:11:40 <skvidal> nirik: cron is actually looking ok 19:11:56 <skvidal> nirik: it breaks out the set of hosts and knows the playbooks to run 19:12:01 <nirik> and then we need to work on moving the builder stuff over. 19:12:03 <nirik> nice! 19:12:20 <skvidal> nirik: I have to 1. parallelize it so we are running them roughly all at the same time (up to a limit) 19:12:27 <skvidal> the trigger thing is hell on wheels 19:12:37 <nirik> yeah, I feared it might be. ;( 19:12:44 <skvidal> it's more or less 'if this is from a task or a playbook directly, great - I can figure that out' 19:12:46 <skvidal> else 'screwed' 19:12:54 <nirik> so, perhaps this: 19:12:55 <skvidal> also - commits to things like vhost-reboot? 19:13:04 <skvidal> not something we want to trigger on :) 19:13:21 <nirik> if its a task/playbook we can easily see what is affected, trigger run 19:13:35 <nirik> if it's a playbook we don't want to do like vhost-reboot, don't do anything. 19:13:46 <skvidal> so isolating the playbooks is easy enough - they are all in a subdir 19:13:46 <nirik> if it's something and we can't tell, touch a file to run cron next time? 19:14:02 <skvidal> nirik: yah - it's what we discussed before 19:14:15 <skvidal> nirik: I was looking at what it would take to include the roles in this whole thing 19:14:17 <nirik> perhaps move the one-time type playbooks to scripts? 19:14:17 <skvidal> like tasks 19:14:24 <nirik> ah yeah, need to get roles in there. 19:14:36 <skvidal> and that's where I kinda meandered down a rathole 19:14:48 <skvidal> nirik: I'm sure you've experienced that. :) 19:14:58 <nirik> yeah. 19:15:05 <nirik> there's many ways to do things... which is great, but... 19:15:09 <skvidal> hah 19:15:27 <nirik> at some point we just need to pick one, do it and then if it sucks redo it 19:15:36 <skvidal> yah - sorry 19:15:41 <skvidal> oh on the builders merge 19:15:48 <skvidal> I put in some group_vars and host_vars for the builders 19:16:05 <skvidal> that are going to be the basis for porting the builders_reinstall playbook 19:16:17 <nirik> cool. 19:16:21 <skvidal> all of it is pretty tame - but I've not been able to test it b/c of the freeze 19:16:34 <skvidal> I really just need one, removed, builder to beat up 19:16:35 <skvidal> :) 19:16:47 <nirik> BTW, I think it might be pretty easy to make a 'arm-reinstall' playbook too... 19:17:01 <skvidal> nirik: can we trigger reboots/ks from lockbox? 19:17:07 <skvidal> nirik: or do we need to delegate to some other box? 19:17:09 <skvidal> (which is fine) 19:17:12 <nirik> yep. or from noc01... 19:17:21 <nirik> via ipmitool commands 19:17:23 <skvidal> how reliable is it? 19:17:37 <nirik> very. I've not had it fail. 19:17:42 <skvidal> cool 19:17:58 <nirik> tell to pxe boot next time, power cycle, wait to finish install 19:18:12 <skvidal> got an average wait? 19:18:17 <skvidal> what's the reinstall time take? 19:18:42 <nirik> a few minutes... but then it takes another 3-4 for sshd to get enough entropy to get ssh host keys 19:18:49 <skvidal> ummm 19:18:52 <skvidal> 3-4 minutes for entropy? 19:18:54 <skvidal> really? 19:18:56 <nirik> or something. 19:18:58 <skvidal> wow 19:19:12 <nirik> I can time it. I might be thinking it's longer than it was 19:19:34 <nirik> they def are up and working but sshd isn't yet for a bit 19:20:03 <nirik> anyhow, lets keep plugging at ansible and try and get some things setup in it/in use... 19:20:12 <nirik> any other sysadmin stuff? 19:20:34 <skvidal> speaking of tihs stuff 19:20:44 <skvidal> at some point I will need to spend more time back on the copr-be 19:20:47 <smooge> need to install entropy network :) 19:21:07 <nirik> skvidal: ok. 19:21:12 <nirik> #topic Private Cloud status update / discussion 19:21:12 <smooge> what does the be stand for skvidal ? 19:21:29 <skvidal> backend 19:21:38 <nirik> so, just a bit of news here, we have been working to reinstall our odd cloudlet and get some of the nodes added to openstack. 19:21:52 <skvidal> 07,05 are done 19:21:53 <nirik> should have that before too long and 2 nodes we can use to test new stuff 19:21:56 <skvidal> 03 is wacked out 19:22:01 <skvidal> 01 is just sitting there 19:22:21 <skvidal> so right now 19:22:29 <nirik> ok. shall we add 05/07 in? 19:22:31 <skvidal> we just need to turn 05/07 into compute nodes 19:22:32 <skvidal> yes 19:22:44 <skvidal> do we have a playbook/processed setup for that? 19:23:03 <nirik> there is one on 02 in /tmp I think, or I have it locally here... 19:23:15 <skvidal> nirik: hmm - might be good to get that somewhere quasi-official 19:23:18 <nirik> it's basically install openstack-nova-compute and copy over the nova.conf file and start it. 19:23:19 <smooge> do the ansible stuff for them run off 01/02 versus lockbox? [silly question I know] 19:23:22 <skvidal> even if that is just in /root on 02 19:23:28 <nirik> skvidal: agreed. 19:23:55 <nirik> I can put it wherever... 19:24:09 <nirik> if someone wants to poke at it, or merge it into our main repo or run it or whatever. 19:25:06 <nirik> ok, any other cloud news? 19:25:12 * skvidal listens to the cricket storm of volunteers 19:25:23 <smooge> sorry you needed something? 19:25:33 <smooge> oh you want it merged with the main ansible 19:25:36 <smooge> I can look at that. 19:25:47 <smooge> sorry 19:25:47 <nirik> cool. 19:25:52 <nirik> #topic Upcoming Tasks/Items 19:26:03 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze 19:26:03 <nirik> #info 2013-04-15 - remove legacy-openid 19:26:03 <nirik> #info 2013-04-15 - announce openid change on hosted. 19:26:03 <nirik> #info 2013-04-15 - drop inactive apprentices 19:26:03 <nirik> #info 2013-04-16 F19 alpha release 19:26:04 <nirik> #info 2013-04-17 - switch to new sigul servers. 19:26:05 <nirik> #info 2013-04-19 - switch hosted to openid 19:26:13 <nirik> anything else people would like to schedule or note or change? 19:26:21 <nirik> oh, alpha has slipped a week, I need to adjust. 19:26:31 <nirik> so, probibly those things after alpha will push out a week 19:27:23 <nirik> #topic Open Floor 19:27:27 <nirik> any items for open floor? 19:28:37 <nirik> Oh... I did whip up some plans on hosted: 19:28:55 <nirik> https://fedoraproject.org/wiki/Infrastructure_FedoraHosted_Version2_Notes 19:29:52 * nirik is sure there's other stuff, but I can't think of it right now. :) 19:30:37 <nirik> ok, if nothing else will close out in a minute. 19:31:36 <smooge> not from me 19:31:51 <nirik> thanks for coming everyone! 19:31:54 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
