============================================ #fedora-meeting: Infrastructure (2012-08-16) ============================================ Meeting started by nirik at 18:00:19 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2012-08-16/infrastructure.2012-08-16-18.00.log.html . Meeting summary --------------- * Greets! (nirik, 18:00:19) * New folks introductions and Apprentice tasks. (nirik, 18:02:16) * Applications status / discussion (nirik, 18:07:29) * LINK: https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13 (nirik, 18:11:03) * comments welcome on websockets/busgateway plan: https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13 (nirik, 18:11:53) * pastebin service going to staging soon. (nirik, 18:12:03) * Sysadmin status / discussion (nirik, 18:19:04) * Hopefully will have networking to our private cloud boxes today and can start setting them up (nirik, 18:19:57) * new machine at osuosl... should be setup hopefully soon. osuosl02 (nirik, 18:20:16) * should have vfilers in the next few weeks to start looking at. (nirik, 18:20:48) * moving hosted03->01/02 next tuesday. (nirik, 18:21:11) * new virthost should be in phx2 now, racked in next few weeks to replace virthost13 (nirik, 18:24:31) * we will be making a busgateway01.stg and paste01.stg soon. (nirik, 18:25:06) * Upcoming Tasks/Items (nirik, 18:26:32) * 2012-08-14 to 2012-08-28 F18 Alpha Freeze (nirik, 18:26:44) * 2012-08-21 hosted03-> hosted01/02 migration (tenative) (nirik, 18:26:44) * 2012-08-28 F18 Alpha release. (nirik, 18:26:44) * 2012-08-31 end of 2nd quarter (nirik, 18:26:44) * 2012-09-01 nag fi-apprentices. (nirik, 18:26:44) * 2012-09-08 drop inactive fi-apprentices. (nirik, 18:26:45) * 2012-09-10 to 2012-09-24 Skvidal is out (nirik, 18:26:47) * 2012-09-18 to 2012-10-02 F18 Beta Freeze (nirik, 18:26:49) * 2012-09-27 to 2012-09-28 Smooge at Puppetconf (nirik, 18:26:51) * 2012-10-02 F18 Beta release (nirik, 18:26:55) * Default Application Frameworks (nirik, 18:28:14) Meeting ended at 19:00:29 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (106) * abadger1999 (48) * skvidal (22) * lmacken (19) * tflink (11) * jds2001 (11) * abompard (11) * threebean (10) * mdomsch (8) * smooge (6) * zodbot (5) * samkottler (4) * relrod (4) * suehle (3) * athmane (1) * rossdylan (1) * ricky (0) * dgilmore (0) * CodeBlock (0) -- 18:00:19 <nirik> #startmeeting Infrastructure (2012-08-16) 18:00:19 <zodbot> Meeting started Thu Aug 16 18:00:19 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:19 <nirik> #meetingname infrastructure 18:00:19 <nirik> #topic Greets! 18:00:19 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 18:00:19 <zodbot> The meeting name has been set to 'infrastructure' 18:00:19 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 18:00:25 * skvidal is here 18:00:41 * samkottler is present 18:00:56 * athmane is around 18:01:02 * abadger1999 here 18:01:04 * abompard is here 18:01:23 * jds2001 around 18:02:00 <nirik> ok, cool. 18:02:07 <nirik> lets go ahead and dive on in. 18:02:16 <nirik> #topic New folks introductions and Apprentice tasks. 18:02:25 <smooge> here 18:02:31 <nirik> If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so in this part of the meeting. Don't be shy! 18:02:49 <abompard> I'm not sure I've been introduced around here 18:03:09 <nirik> abompard: nope... please do introduce yourself. ;) 18:03:14 <abompard> some of you got an email from spot, but maybe not everybody 18:03:30 <abompard> I'm a new hire in the Fedora Engineering team at Red Hat 18:03:48 <nirik> Welcome abompard. 18:03:50 <abompard> I'll be working on HyperKitty first, the next Mailman 3 archiver 18:03:56 * samkottler claps :) 18:03:57 <abompard> thanks nirik 18:04:17 <nirik> do let us know if you need anything or are interested in working on other things too... 18:04:23 <abompard> I'm in Paris, it's UTC+2 so it's 8PM here 18:04:35 <abompard> nirik: I will, there's a lot of cool stuff to do 18:04:48 <abompard> nirik: but one thing at a time (well, not one, but not too many) 18:04:49 <nirik> yeah, we aren't lacking for stuff to work on. ;) 18:05:16 <nirik> excellent. Look forward to working with you... 18:05:26 <abompard> I hope I'll be able to attend the Thursday meetings 18:05:34 <nirik> any other new folks? or general apprentice queries? 18:05:42 * relrod is here. 18:05:51 * rossdylan is here 18:05:57 <nirik> abompard: if you can't and you have something to bring up, feel free to send it to the list. 18:06:07 <abompard> nirik: got it 18:06:12 * mdomsch 18:06:20 * lmacken 18:06:25 <nirik> cool. welcome everyone, good turnout today. ;) 18:06:28 <samkottler> I spoke to a few different people about our pupet setup so I'll be doing more work on that in the coming weeks 18:06:29 * threebean 18:06:37 <samkottler> No query, though :) 18:07:16 <nirik> samkottler: great. Thanks for updating our puppet docs too 18:07:27 <nirik> ok, moving along... 18:07:29 <nirik> #topic Applications status / discussion 18:07:39 <nirik> any application news this week or upcoming? 18:07:49 <nirik> we are in freeze, so that means not much deployment... 18:08:24 <threebean> abadger1999, fchiulli, and I are working on migrating some pieces of pkgdb into the tagger and packages apps. Still in development now, but we hope to have some chunks done by the end of the freeze. 18:08:48 <nirik> I'm going to try and help track down our pkgdb hotfixes too. 18:08:57 <nirik> so we can make sure all those are merged. 18:08:58 <abadger1999> <nod> bugz seemed almost implemented in packages already so that should be done and ready to deploy after freeze. 18:09:28 <abadger1999> other stuff may require more work as it seems packages doesn't keep all that data yet. 18:09:35 <nirik> if anyone wants to help with that, I'd be happy to divide and conquer. 18:10:42 <nirik> I updated our websockets/busgateway ticket. 18:10:50 <nirik> with a plan... if anyone has comments, please let me know soon. 18:11:03 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13 18:11:35 <nirik> also, in application news, looks like our pastebin service is ready to move to staging... 18:11:53 <nirik> #info comments welcome on websockets/busgateway plan: https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13 18:12:03 <nirik> #info pastebin service going to staging soon. 18:12:29 <nirik> any other apps news? 18:12:41 <threebean> Anyone mind if I start monkeying with koji in stg? 18:12:54 <lmacken> new bodhi went out, last week, and I pushed out some additional bugfixes earlier this week 18:13:16 <nirik> lmacken: did those tickets get closed? we should do that if not... 18:13:27 <lmacken> fedora-packages indexer got wedged again as well.. not sure what is going on there, but I kicked it in the mean time. 18:13:33 <nirik> threebean: go for it. It has a db load in it... but I don't think it has builders or anything. 18:13:44 <lmacken> nirik: ah, I'll close them out, thanks 18:14:17 <nirik> threebean: but if you need a builder, perhaps skvidal can whip you up one on pretty short notice. 18:14:28 <skvidal> yah 18:14:29 <threebean> nirik: Cool. I'm in the dark as to how it actually works. Going to try out some fedmsg stuff.. I might end up needing a builder to test it all. 18:14:30 <skvidal> I can do things 18:14:32 <skvidal> :) 18:14:40 <threebean> skvidal: :) 18:14:46 <threebean> skvidal: I will bother you, then. 18:14:54 <skvidal> threebean: as long as its easy. :) 18:14:57 <nirik> it would be nice for it to have a builder and also build from pkgs01.stg git, etc. 18:15:11 <nirik> because then we could actually test mock updates and such 18:15:33 <jds2001> yeah, i'd like for the vendor in the koji config to be different 18:15:45 <jds2001> such that you *know* that it came from staging without a dobut. 18:15:51 <nirik> yeah, that would be good too. 18:16:16 <nirik> and... (wait for the ponies and rainbows)... 18:16:42 <nirik> it would be cool if it ran koji-shadow and shadowed the main koji. Then we could see if new mock breaks something much more naturally 18:17:05 * jds2001 wonders where you plan on finding hte disk for that :D 18:17:32 <nirik> dunno. it doesn't need to follow all tags... rawhide can't be that big can it? ;) 18:17:56 <nirik> anyhow, those are all dreams. If someone wants to make them real I will buy them a drink. ;) 18:18:18 <jds2001> it'd take a lot more than drinks, i think :D 18:18:23 <nirik> ok, any other app news or shall we move on? 18:18:31 <relrod> FAS is in string freeze now 18:18:40 <nirik> relrod: cool. :) 18:18:52 <relrod> I'm waiting to hear back on one ticket then I'll email trans@ and they can get to work 18:19:00 <nirik> sounds good. 18:19:04 <nirik> #topic Sysadmin status / discussion 18:19:21 <nirik> so, we are in freeze so that slows down activity (hurray! :) 18:19:36 <nirik> however there's various bits of news: 18:19:57 <nirik> #info Hopefully will have networking to our private cloud boxes today and can start setting them up 18:20:16 <nirik> #info new machine at osuosl... should be setup hopefully soon. osuosl02 18:20:48 <nirik> #info should have vfilers in the next few weeks to start looking at. 18:21:11 <nirik> #info moving hosted03->01/02 next tuesday. 18:21:35 <nirik> Anything else pending or upcoming? 18:22:09 <nirik> skvidal wrote up a wiki page on our ansible pokings so far: https://fedoraproject.org/wiki/User:Skvidal/Ansible 18:22:46 <abadger1999> skvidal: Do we need simplejson? We have python-2.6 on all boxes except app07. 18:22:51 <skvidal> abadger1999: we don't 18:22:54 <abadger1999> Cool. 18:22:56 <skvidal> abadger1999: it's just *json( 18:23:01 <skvidal> but rhel5 needs it iirc 18:23:06 <abadger1999> Yeah. silly rhel5 18:23:16 <skvidal> I wanted to be reasonably exact 18:23:20 <skvidal> so no one nitpicked :) 18:23:24 <abadger1999> <nod> 18:23:31 <skvidal> pretty much @core gets it done 18:24:02 <skvidal> I'm working on breaking the repo I use for builders now 18:24:09 <skvidal> up into private and public 18:24:11 <skvidal> so I can export ours 18:24:15 <nirik> oh, another one: 18:24:31 <nirik> #info new virthost should be in phx2 now, racked in next few weeks to replace virthost13 18:24:38 <jds2001> skvidal: huh? 18:24:47 <skvidal> jds2001: our git repo for builder config 18:24:55 <jds2001> oh oh 18:25:00 <skvidal> jds2001: where I'm storing ansible playbooks and config files, etc 18:25:05 <jds2001> i thought you meant yum repo :) 18:25:06 <nirik> #info we will be making a busgateway01.stg and paste01.stg soon. 18:25:57 <nirik> ok, any other sysadmin type things to note? 18:26:32 <nirik> #topic Upcoming Tasks/Items 18:26:42 <nirik> here's the list I have of upcoming: 18:26:44 <nirik> #info 2012-08-14 to 2012-08-28 F18 Alpha Freeze 18:26:44 <nirik> #info 2012-08-21 hosted03-> hosted01/02 migration (tenative) 18:26:44 <nirik> #info 2012-08-28 F18 Alpha release. 18:26:44 <nirik> #info 2012-08-31 end of 2nd quarter 18:26:44 <nirik> #info 2012-09-01 nag fi-apprentices. 18:26:45 <nirik> #info 2012-09-08 drop inactive fi-apprentices. 18:26:47 <nirik> #info 2012-09-10 to 2012-09-24 Skvidal is out 18:26:49 <nirik> #info 2012-09-18 to 2012-10-02 F18 Beta Freeze 18:26:51 <nirik> #info 2012-09-27 to 2012-09-28 Smooge at Puppetconf 18:26:55 <nirik> #info 2012-10-02 F18 Beta release 18:26:57 <nirik> anyone have items they would like to add/note/remove/edit? 18:27:22 <smooge> May want ot make that the 26th so I can travel to SF :) 18:27:31 <nirik> oh yeah. ;) 18:27:49 * nirik edits. 18:28:00 <nirik> ok, anything else on schedules? 18:28:14 <nirik> #topic Default Application Frameworks 18:28:24 <nirik> we didn't reach any conclusions on this the other week... 18:29:00 <nirik> anyone care to propose something? or ? 18:29:56 <tflink> I have a potentially related question about flask packages in epel6 18:30:14 <nirik> I guess I tried to propose: flask, TG2, pyramid. But abadger1999 didn't like that due to too many, and lmacken didn't like that due to too few? 18:30:22 <tflink> assuming that we do end up using Flask more, is there a version that we're trying to target? 18:30:29 <nirik> tflink: ask away... relrod and ianweller are leading that effort. 18:30:34 <abadger1999> yeah -- really.. I'd like 1 framework. 18:30:41 <tflink> I ask because there are a bunch of clask extension packages in epel6 that can't play nice together 18:30:46 <tflink> s/clask/Flask 18:30:56 <lmacken> I think instead of just trying to bless an arbitrary number of frameworks, I think we should take a step back and define the critera/policy of frameworks that we wish to allow. Maybe only 1 meets them, maybe 5? 18:30:59 <abadger1999> proposing 2 was already a compromise to what I was hoping for. 18:31:21 <tflink> it looks like the packaging effort for the extensions was started and not quite finished 18:31:44 <abadger1999> lmacken: That might be logical to think about if our concern is new development... my concern is maintainance, though. 18:32:01 <abadger1999> I think from that standpoint, the answer is: pick one and stick with that. 18:32:51 <lmacken> if maintenance of our existing apps is the main concern, then rewriting all of them completely in something that we've never had experience deploying may not be the best choice. 18:32:57 <jds2001> abadger1999: but the answer could be that we have people able/willing to maintain 5 18:33:09 <nirik> things I want: fewer parts/deps/changes, active upstream that answers bugs/security issues, not too rapid updates but updates, more developers that understand it so can fix things/track down bugs 18:33:14 <abadger1999> lmacken: then let's settle on TG2 and port everything to that. 18:33:26 <abadger1999> jds2001: right now -- but that has many problems 18:34:15 <abadger1999> Person A maintains Foo written in framework1; person B maintains Bar, Baz, and five other apps written in framework1. 18:34:25 <relrod> tflink: I don't know much about the extensions yet, I've just been focusing on getting Flask itself in EPEL for now 18:34:32 <abadger1999> Person A has done the heavy lifting of packaging, upstream interfacing etc... then they go away. 18:34:43 <lmacken> abadger1999: I'm not against porting/rewriting FAS & friends in TG2. But I'm also not against someone wanting to do it with pyramid, flask, etc. 18:34:44 <abadger1999> Person B doesn't have time to maintain the framework... 18:34:57 <tflink> relrod: I have been working on updating the specs and building locally for now, I was planning to pester the maintainer soon for an update 18:35:11 <nirik> perhaps we need to more strongly cross train people? 18:35:58 <nirik> ie, personA: you maintain bar this week and personB: you maintain foo this week? 18:35:59 <abadger1999> * New user joins infra and starts work on Baz written to framework 1. Makes changes that mean porting app Foo written to framework 2.... they have to learn both frameworks to get their change in. 18:36:50 <nirik> abadger1999: there's still some issue even if everything is the same framework... 18:36:58 <nirik> perhaps less so tho 18:37:10 <abadger1999> nirik: of course... but we partially mitigate them. 18:37:31 <jds2001> person A writes app Foo and goes away, Foo is now dead :( 18:37:32 <abadger1999> so we're in a better place even if we're not in a perfect place. 18:37:41 <jds2001> doesnt much matter what framework it's in 18:37:42 <tflink> I've heard the argument that a lot of the learning has to do with the application on top of the framework - learning a framework is easy. learning all the project-specific quirks and oddities is the hard part 18:38:15 <tflink> but that's developer learning curve, not maintaining tons of different frameworks 18:38:34 <abadger1999> jds2001: That can happen too. But let's say some app were to be abandoned that is critical to Fedora (packagedb, mirrormanager, bodhi, koji... one of those). 18:38:39 <smooge> my take on it more is from the non-developer sysadmin. 18:39:12 <abadger1999> jds2001: is it easier for someone else who works on one of those other apps to step in and start working if the app they currently maintain runs on that framework? Or if it's totally foreign? 18:39:47 <abadger1999> My experience contributing patches to any of those TG apps vs staring at koji code is that a common framework is huge boost to getting into the code base. 18:40:12 <nirik> is koji even using a framework? 18:40:21 <lmacken> nope 18:40:24 <abadger1999> is raw wsgi a framework? ;-) 18:40:34 <nirik> yeah, that. 18:41:08 <nirik> if we said: all applications must use flask. Would that mean someone would have to re-write koji in it? or askbot? or mediawiki? :) 18:41:19 <abadger1999> I could have said something like ask.fedoraproject.org (django).. same thing applies. 18:41:39 <abadger1999> nirik: those three we currently treat as third party apps 18:41:48 <abadger1999> we run them but we realy don't develop them. 18:41:49 <nirik> right. 18:41:53 <abadger1999> nirik: also -- 18:42:21 <abadger1999> this applies most immediately to new app development. 18:42:29 <nirik> anyhow, are we going to reach a consensus here? or perhaps we just settle on the 3 and leave it at that until/unless someone makes a good argument for removing one or adding a new one? 18:42:31 <abadger1999> new apps should be written in our chosen framework(s) 18:42:35 <nirik> right 18:42:39 <abadger1999> old apps get ported as we have the necessity. 18:42:45 <abadger1999> TG1 is becoming a necessity. 18:42:58 <abadger1999> But TG2 , pyramid.... not so much. 18:43:12 <lmacken> if I could leave bodhi as TG1 for 5 more years, I totally would. 18:43:30 <nirik> but 2.0 needs newer framework? or ? 18:43:33 <abadger1999> lmacken: yep.... but hte TG1 stack becomes harder to support every year. 18:43:35 <lmacken> there's something to be said about a framework that we've had in production for 5 years already, with very few problems at all. 18:43:45 <abadger1999> yeah tg2 has different underpinnings. 18:43:56 <lmacken> nirik: yeah, I designed myself into a corner with the architecture of bodhi1, it wasn't TG1's fault 18:44:17 <nirik> lmacken: live and learn. ;) I bet that code looks fun to you now... 18:44:29 <lmacken> yeah, it's pretty horrid :) 18:45:14 <abadger1999> If only TG had gone with evolutionary development from tg1.0 instead of rearchitechting the stack for tg2... I think we'd all still be happily running tg1 :-) 18:45:16 <lmacken> but I think we're going to keep going through that live & learn cycle with other frameworks, over and over. We just need to keep moving forward without making the same mistakes, which will be hard if we deviate too far from where we've been 18:45:36 <abadger1999> but htey didn't... and tg1 is much less healthy upstream than the other choices. 18:46:48 <smooge> ok quick question.. 18:47:16 <lmacken> if we're looking to make strong recommendations as to what frameworks we wish to use in the future, again, I think we need to decide upon critera that they must meet 18:47:55 <smooge> if lmacken or abadger won the lottery tomorrow and we didn't have them.. how tenable are our apps currently? 18:48:06 <lmacken> like, a) active upstream b) active fedora maintainers ... 18:48:12 <threebean> smooge: I've got 'em. 18:48:13 <lmacken> smooge: I came close last night ;) 18:48:20 <nirik> a) active upstream b) active epel maintainer(s) c) something we already use d) your need here 18:49:12 <threebean> smooge: honestly, pkgdb and mirrormanager appear to be the weakest links from my PoV. The rest could be spread around. 18:49:24 <abadger1999> smooge: We wouldn't die for Fedora 18 but we'd be starting to hurt by fedora 19 release. 18:50:23 <smooge> ok 18:50:29 <abadger1999> smooge: hotfixes and new releases with those hotfixes are the two places you'd notice it first. 18:50:44 * nirik notes as tflink pointed out there are still things we need to address for flask. Likely we won't know all those until we have something deployed using it. 18:51:01 <abadger1999> I generate a lot of hotfixes for fas, python-fedora, and pkgdb. And as time permits... I even wrap those up into releases :-) 18:51:14 <tflink> nirik: sounds like I might be the guinea pig there 18:51:16 <threebean> nirik: the exact same applies for pyramid. 18:51:38 <lmacken> threebean: well, I have experience with pyramid in our env now with openhw2012 18:51:40 <nirik> yeah, true... and we are still on the early part of the curve for TG2 18:51:52 * mdomsch needs to get MM 1.4 out the door, but I've been saying that for 18 months... 18:52:02 <lmacken> threebean: and now we know that the PrefixMiddleware is crucial :) 18:52:02 <abadger1999> the same could apply for any framework we have... but we don't want to deploy one app in each of them to see how we like it :-) 18:52:14 <nirik> mdomsch: is that just a new bugfix release? or rewritten in tg2? 18:52:23 <nirik> abadger1999: ageed. 18:52:26 <nirik> agreed even 18:52:29 <mdomsch> nirik: bugfix/features, not rewritten 18:52:35 <tflink> abadger1999: don't we already pretty much have one in each already? 18:53:08 <lmacken> we're pretty close. I think all we need is web2py in the mix and we'll have them all :P 18:53:10 <abadger1999> tflink: not that I know of -- we have tg1 and tg2 in production. 18:53:24 <abadger1999> tflink: we have pyramid and flask being worked on (pre-stg still I think) 18:53:34 <tflink> ah, I wasn't thinking of production. just the writing of the apps 18:53:37 <abadger1999> We have django third party apps deployed. 18:53:42 <nirik> I guess from the sysadmin side this would be similar to oses... and there we are pretty much standardized. Think of "I'd like to deploy $thing, but it uses SuSE" :) 18:54:36 <abadger1999> But I think if we wrote our own django apps we'd have a better understanding of rhe framework and that would likely lead to changes in how we deployed them. 18:54:47 <skvidal> okay 18:54:56 <skvidal> I've been kinda quiet for a bit - I wanted to interject 18:55:01 * nirik sees time running low. 18:55:04 <nirik> by all means skvidal 18:55:11 <abadger1999> So yeah -- there are things we won't know until for real deployment -- but we don't want to really deploy to find them out. 18:55:11 <skvidal> should we stop thinking of the model of how we are currently deploying apps 18:56:09 <skvidal> and start thinking about modifying our deployments to make it less of an effort to deploy one-off stacks 18:56:42 <mdomsch> skvidal: the tricky integration is with FAS 18:56:47 <nirik> but this isn't really about one-off is it? 18:57:21 <mdomsch> at least for MM, that's the one bit outside of the framework that abadger1999 has had to make functional 18:57:34 <mdomsch> which begs the question of the future of FAS as an integration point 18:57:39 <abadger1999> yeah -- it's not really about one-off. It's about maintainance more so. I think one-off stacks would make long term maintanance even harder? 18:57:39 <skvidal> mdomsch: you're right - that is the tricky bit 18:58:03 <mdomsch> I'd be OK with one-off stacks if there were a) fewer integrations; b) more hands to help 18:58:14 <skvidal> mdomsch: +1 18:58:48 <abadger1999> since one-off stacks could be used for pegging versions.. but that means needing to maintain more overall combinations of things with those pegged versions. 18:58:52 <mdomsch> if somehow we went from python-fedora FAS integration to "some common account API" world, be it FAS of LDAP or Shibboleth or AD or ... 18:59:48 <zodbot> suehle: Error: Can't start another meeting, one is in progress. 18:59:51 <suehle> doh 19:00:02 <nirik> suehle: sorry if we are intruding on your time... 19:00:02 * suehle should learn to read topic, sorry 19:00:08 <nirik> shall we wrap up folks? 19:00:12 <abadger1999> Sure. 19:00:16 <nirik> continue in #fedora-apps? 19:00:17 * threebean nods 19:00:22 <suehle> thanks! 19:00:26 <nirik> Thanks for coming everyone! 19:00:29 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
