============================================ #fedora-meeting: Infrastructure (2012-05-24) ============================================ Meeting started by nirik at 18:01:02 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2012-05-24/infrastructure.2012-05-24-18.01.log.html . Meeting summary --------------- * Robot Roll Call (nirik, 18:01:02) * New folks introductions and Apprentice tasks. (nirik, 18:03:37) * bugzilla and database issues this week (nirik, 18:08:33) * ACTION: nirik to file outage tickets for these two things so we have record of them. (nirik, 18:20:26) * plan to move fas db to it's own db server. (nirik, 18:20:40) * adjust freeze document (nirik, 18:20:51) * two factor auth status (nirik, 18:23:09) * Fedora 17 Release tickets (nirik, 18:24:25) * release 2012-05-29 (nirik, 18:24:45) * LINK: https://fedorahosted.org/rel-eng/ticket/4963 (nirik, 18:30:08) * Applications status / discussion (nirik, 18:31:57) * LINK: http://findingscience.com/mod_auth_openid/ if anyone wants to look it over. (nirik, 18:40:51) * Upcoming Tasks/Items (nirik, 18:42:16) * 2012-05-08 to 2012-05-29 FINAL FREEZE (nirik, 18:42:16) * 2012-05-29 - F17 release (nirik, 18:42:16) * 2012-06-01 - nag fi-apprentices. (nirik, 18:42:16) * 2011-06-03 - gitweb-cache removal day. (nirik, 18:42:16) * 2012-06-04 - class B reboots? (nirik, 18:42:16) * 2012-06-05 - class A reboots? (nirik, 18:42:17) * 2012-06-08 OOW: osuosl01.fedoraproject.org (nirik, 18:42:19) * 2012-06-17 OOW: sign-vault02.phx2.fedoraproject.org (nirik, 18:42:20) * 2012-06-21 to 2012-07-04 Kevin is off on trains and boats. (nirik, 18:42:22) * sometime in June hardware will show up in Phx for new systems (smooge, 18:44:44) * mediawiki 119 upgrade coming. (nirik, 18:51:02) * koji upgrades coming (nirik, 18:51:09) * will apply fix for mod_wsgi hash stuff soon (nirik, 18:51:25) * trip out onsite to phx2 sometime in the next few months. (nirik, 18:51:41) * mirrormanager 1.4 coming too (nirik, 18:52:23) * Open Floor (nirik, 18:52:56) * LINK: https://aws.amazon.com/amis (skvidal, 19:01:02) Meeting ended at 19:02:51 UTC. Action Items ------------ * nirik to file outage tickets for these two things so we have record of them. Action Items, by person ----------------------- * nirik * nirik to file outage tickets for these two things so we have record of them. * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * nirik (140) * skvidal (34) * dgilmore (33) * mdomsch (31) * marcdeop (18) * abadger1999 (10) * zodbot (10) * cyberworm54 (10) * relrod (10) * lmacken (9) * pingou (8) * smooge (5) * threebean (4) * codemaniac (2) * inode0 (2) * jds2001 (2) * fcami__ (1) * ricky (0) * CodeBlock (0) -- 18:01:02 <nirik> #startmeeting Infrastructure (2012-05-24) 18:01:02 <nirik> #meetingname infrastructure 18:01:02 <nirik> #topic Robot Roll Call 18:01:02 <zodbot> Meeting started Thu May 24 18:01:02 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:01:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:01:02 <zodbot> The meeting name has been set to 'infrastructure' 18:01:02 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 18:01:02 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 18:01:30 * abadger1999 here 18:01:36 * lmacken 18:01:38 * threebean is here 18:01:42 * skvidal is here 18:01:47 <dgilmore> hola 18:01:51 * jds2001 around 18:02:13 <relrod> here 18:02:32 <cyberworm54> hello evrybody 18:02:57 <nirik> welcome everyone. 18:03:33 <nirik> ok, lets go ahead and dive in then 18:03:37 <nirik> #topic New folks introductions and Apprentice tasks. 18:03:37 <nirik> If any new folks want to give a quick one line bio or any apprentices 18:03:37 <nirik> would like to ask general questions, they can do so now. Anyone? 18:03:56 <cyberworm54> i will 18:04:09 * marcdeop is sorry to be late 18:04:20 <nirik> no worries. 18:04:32 <cyberworm54> My name is Ivan garcia I've been a linux kvm admin for 4 years now 18:04:50 <cyberworm54> living in the USA 18:05:18 <nirik> cyberworm54: welcome. :) Are you most interested in the sysadmin side of things? or application devel/programming? 18:06:10 <cyberworm54> thank you :) I am more in the infrastructure side/sysadmin, I have proggramming skills and scripting too such as python and bash 18:06:27 * codemaniac is here 18:06:35 <nirik> great. Come see me in #fedora-admin later this afternoon and we can see about getting you setup in the apprentice group... 18:06:42 <nirik> welcome codemaniac 18:07:00 <codemaniac> \o nirik .Good to see you . 18:07:03 <nirik> so, any other general questions from apprentices or new folks? Or shall we move on? 18:07:08 <cyberworm54> that would be excellent I am really excited about this opportunity :) thank you 18:07:37 * marcdeop welcomes the new fellow! 18:08:09 <cyberworm54> :) 18:08:17 <cyberworm54> thanks marcdeop 18:08:17 <nirik> ok, I'd like to diverge from the agenda I sent out for a bit and talk about issues we have been hitting this week... 18:08:33 <nirik> #topic bugzilla and database issues this week 18:08:53 <nirik> So, the first issue we ran into was that bugzilla.redhat.com was upgraded last saturday. 18:09:14 <nirik> We (in retrospect foolishly) updated python-bugzilla on our servers. 18:09:47 <nirik> The new python-bugzilla has some behavior and cases where it causes the same query as before to generate a LOT of additional queries with the new interface 18:10:11 <nirik> We have now downgraded back to the old stable python-bugzilla and (almost) everything should be back to normal. 18:10:28 <nirik> What can we do to better avoid this kind of thing moving forward? 18:11:07 <nirik> One thing I can think of is to better test in stg with partner-bugzilla. 18:11:21 <skvidal> nirik: to be fair 18:11:25 <nirik> Red Hat runs a bugzilla instance at partner-bugzilla.redhat.com that has email turned off. 18:11:30 <skvidal> nirik: the limited scale of tests in staging 18:11:30 <marcdeop> easnt here. tezting bugzilla for some weeks? 18:11:39 <skvidal> might not have tipped off the issues 18:11:47 <skvidal> and the load issues may not have been visible 18:12:03 <nirik> skvidal: true. Also, we didn't know until RHIT contacted us and blocked us out for causing too much load. ;) 18:12:08 <skvidal> nirik: indeed 18:12:37 <nirik> marcdeop: they updated the testing one a while back, but only updated the production one (bugzilla.redhat.com) last saturday 18:13:12 <nirik> Anyhow, I will probibly send a email on this to the list too with more info about what happened, etc. 18:13:31 <nirik> Right on the heels of fixing the bugzilla issues, we ran into some database problems. 18:13:39 * pingou (late) 18:13:55 <nirik> This was caused we think by a fedora-tagger update that hit the database hard every few hours. 18:14:25 * fcami__ is here (sorta) 18:14:26 <marcdeop> nirik: were those performance problems? 18:14:50 <skvidal> marcdeop: ultimately, that's what we _think_ it was 18:14:51 <nirik> marcdeop: yeah, worse, it would fill up all our database connections so our account system couldn't authenticate anyone. 18:15:10 <marcdeop> nirik: we could check our database connections 18:15:15 <marcdeop> with nagios 18:15:21 <marcdeop> and warn us in a threshold 18:15:37 <nirik> marcdeop: yeah, but sadly when the account system goes down, that means ALL our web apps that need it to authenticate go down. 18:15:50 <nirik> So, nagios was sending us about 150-200 emails anytime this happened. 18:16:01 <nirik> another nagios check isn't going to help, unless it reduces that. :) 18:16:17 <marcdeop> well,I was thinking more about *identifying* the problem 18:16:24 <marcdeop> as you remarked the work _think_ 18:16:32 <marcdeop> s/work/word 18:16:37 * nirik is still not 100% sure that was the problem, but reverting that seems to have fixed it. 18:16:56 <skvidal> marcdeop: the issue looks like a tagger/packages update beating up db connections 18:17:10 <skvidal> marcdeop: we're taking multiple steps right now to help solve this for the future 18:17:16 <skvidal> 1. the rollback is a temp solution 18:17:29 <skvidal> 2. moving fas' db to its own instance - away from everything else 18:17:54 <mdomsch> when I get MM 1.4 solidified, it'll be much nicer to its database. Still needs a boatload of connections for the crawler though 18:17:57 <nirik> yeah, currently out account system db is on the same db host with other things that may be less important 18:18:14 <skvidal> marcdeop: I agree that figuring out what was triggering it is a good idea - but moving fas db is important independent of any failures from today 18:18:24 <skvidal> today just makes it acute, rather than a chronic concern 18:18:30 <nirik> 3. increasing what is in the freeze, so we don't make changes like this that might affect the larger picture. 18:18:32 <marcdeop> skvidal: that's right 18:19:03 <mdomsch> skvidal: +1 18:19:13 <mdomsch> nirik: we can't control RH IT with our freezes 18:19:15 <dgilmore> skvidal: indeed 18:19:28 <nirik> mdomsch: very true... 18:19:39 <nirik> but we could have not been updating packages/tagger now. 18:20:04 <skvidal> I just posted a freeze break request to infra list - +1's for the general process and I'll post more for the puppet changes 18:20:26 <nirik> #action nirik to file outage tickets for these two things so we have record of them. 18:20:40 <nirik> #info plan to move fas db to it's own db server. 18:20:44 <pingou> nirik: have we actually had outage ? 18:20:51 <nirik> #info adjust freeze document 18:20:57 <pingou> or were we quick enough to reboot the pg server every two hours ? 18:20:59 <nirik> pingou: well, depends on how you define it. 18:21:18 <nirik> there's a short window where things were not authenticating. (less than a minute I'd guess) 18:21:20 <pingou> nirik: let's say outage longer than few min 18:21:24 <pingou> ok 18:21:34 <nirik> also, for the bugzilla thing, maintainers couldn't add bugs to their updates... 18:21:42 <nirik> and bugs.fedoraproject.org didn't work 18:22:34 <nirik> so, it was sorta kinda an outage of not so important things. 18:22:53 <nirik> anyhow, just wanted to talk about those items real quick... will move along now. 18:22:56 * marcdeop wonders if bugs works as of right now 18:23:04 <nirik> marcdeop: it should, yeah 18:23:09 <nirik> #topic two factor auth status 18:23:18 <nirik> any news on the two factor front this week? 18:23:40 <skvidal> nope 18:23:42 <skvidal> next 18:23:47 <skvidal> actually.. 18:23:51 <skvidal> I haven't seen wolfkit in a bit 18:24:21 <nirik> yeah, everyone is busy. :) No worries... 18:24:25 <nirik> #topic Fedora 17 Release tickets 18:24:36 <nirik> we have release tickets lined up for the release on tuesday. 18:24:45 <nirik> #info release 2012-05-29 18:25:07 <nirik> .ticket 3285 18:25:08 <zodbot> nirik: #3285 (Fedora17 Final - new website) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3285 18:25:11 <nirik> .ticket 3286 18:25:16 <zodbot> nirik: #3286 (Fedora17 Final: verify mirror space) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3286 18:25:21 <nirik> .ticket 3287 18:25:23 <zodbot> nirik: #3287 (Fedora17 Final - release day ticket) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3287 18:25:29 <nirik> .ticket 3288 18:25:30 <zodbot> nirik: #3288 (Fedora17 Final - verify permissions on content) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3288 18:25:34 <nirik> .ticket 3289 18:25:35 <zodbot> nirik: #3289 (Fedora17 Final - modify Template:FedoraVersion on wiki) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3289 18:25:43 <nirik> .ticket 3290 18:25:45 <zodbot> nirik: #3290 (Fedora17 Final - update stats gathering scripts for new release) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3290 18:25:56 <nirik> I think we are pretty much on track with them all. 18:26:08 <nirik> as usual, we can't check bits until rel-eng stages them 18:26:28 <mdomsch> nirik: any guess when that is? 18:26:35 <mdomsch> today or tomorrow? 18:26:40 <nirik> dgilmore: ^ ? :) 18:26:53 <dgilmore> nirik: ill be staging today 18:26:55 * mdomsch needs to be sure the move-to-release script doesn't blow up again like it did last time 18:26:59 <dgilmore> and disabling buildbranched 18:27:08 <nirik> great. 18:27:50 <dgilmore> likely ill flip the bits for the mirrors today, but it might be in the morning 18:27:57 <nirik> smooge: you set to take the verify ones you usually do? or would you like someone else to? 18:28:08 <dgilmore> going to tripple check that hardlinking etc is right before opening up 18:28:29 <smooge> I am ready 18:28:34 <nirik> dgilmore: are we nuking drpms? I think there was a request to do that last cycle? 18:28:48 <smooge> I will start checking after he has staged. We have plenty fo disk space still 18:29:13 <dgilmore> nirik: i dont know 18:29:22 <dgilmore> nirik: it feels like we shouldnt nuke it 18:29:47 <nirik> they don't do much good 18:30:08 <nirik> https://fedorahosted.org/rel-eng/ticket/4963 18:30:11 <dgilmore> no they dont 18:30:14 <nirik> anyhow, just thought I would mention it. 18:30:20 <nirik> anything else for release? 18:30:22 <dgilmore> and skvidal swears that yum wont break if they dont exist 18:30:30 <skvidal> what did I do? 18:30:33 <skvidal> drpms? 18:30:41 <skvidal> oh 18:30:42 <skvidal> yah 18:30:47 <mdomsch> that's save a few GB 18:30:47 <dgilmore> skvidal: if we drop drpms from Everything tree 18:30:57 <skvidal> pretty confident in the opportunistic nature of yum on that 18:31:05 <skvidal> well 18:31:07 <skvidal> mostly confident 18:31:08 <skvidal> :) 18:31:18 <dgilmore> skvidal: its the mostly that scares me 18:31:21 <skvidal> :) 18:31:25 <skvidal> LIVE IN FEAR 18:31:33 <marcdeop> die in peace! 18:31:38 <marcdeop> :P 18:31:47 <nirik> anyhow, shall we move along... 18:31:53 <dgilmore> yes 18:31:57 <nirik> #topic Applications status / discussion 18:32:09 <relrod> ooh me me pick me :) 18:32:13 <relrod> So the fedorahosted app I've started on is coming along well, threebean has helped a lot too :) It's moved to Fedora Hosted now (mirrored to github still): https://fedorahosted.org/fedorahosted/. I did my first test of things on a RHEL 6 vm last night, the CLI side of the app works fine, but python-flask isn't packaged for RHEL. 18:32:15 <nirik> lmacken / abadger1999 / threebean / pingou / relrod: any apps news? 18:32:18 <relrod> *so* my question is this. The web app has no tie to anything Fedora Hosted right now. It's just a web form and a db that stores stuff, that may end up having FAS auth so people can't create requests for $random_fas_username. Do we want to explore putting and leaving the web frontend on openshift? Or do we want to get Flask +deps in EPEL? I'm not sure what the general FI stance is, re: "production-ish" stuff on openshift. 18:32:26 * relrod may have had that typed out beforehand ;) 18:32:41 * lmacken did lots of fedora-packages/moksha work this week... nothing visible though. 18:32:59 <nirik> I'd perfer things like that we do host ourselves... just to avoid an external dependency. 18:33:20 <lmacken> apps.fp.o/{packages,tagger} are pretty much done though. 18:33:28 * threebean prepares for the python-flask-* packaging party 18:33:35 <relrod> nirik: ok. It was just a thought 18:33:40 * dgilmore has tasks for a webapp developer if they want something to do 18:33:51 <nirik> relrod: it looks like it has a epel branch, but just was never built for some reason. 18:34:27 * skvidal has been working on a coprs-submission front end using flask and openshift, too - it's just a test/wip 18:34:34 <nirik> lmacken: thats cool. Do we want to do some kind of announcement? perhaps after release? 18:34:40 <pingou> I got busy last week-end on mongodb vs postgresql for HyperKitty, but not much since 18:34:46 <lmacken> nirik: yeah, that sounds good. 18:35:22 <lmacken> nirik: I still need to track down why some new packages aren't getting indexed properly, so holding off on the apps.fp.o announcement until after the release is probably a good call. 18:35:25 <pingou> ah kinda fix fedora-active-user to the new bz :) 18:35:51 <nirik> lmacken: yeah, and that way we can doublecheck over things to make sure it's all set. 18:35:56 <nirik> without freeze issues, etc 18:36:22 <relrod> then to that end (re: packaging python-flask-*) if someone is bored and wants to get these in EPEL, I would love you: python-flask, python-flask-sqlalchemy, python-flask-wtf, python-wtforms 18:36:43 <threebean> relrod: thinking -- we should hammer out the FAS auth for flask but separate it out once we're done into python-fedora-flask 18:37:00 <abadger1999> <nod> 18:37:00 * threebean wonders if ianweller's fudcon app has already done this 18:37:07 <relrod> his app uses openid 18:37:08 <abadger1999> I think it uses openid 18:37:09 <skvidal> threebean: I was planning on just using openid + limited to fedoraproject openid only 18:37:11 <nirik> relrod: they appear to have epel branches. I'd try and contact the maintainer and see if they mind you managing those since they never built for it. 18:37:22 <skvidal> abadger1999: is there a good reason to NOT do that? 18:37:50 <abadger1999> skvidal: I like using openid rather than dedicated fas auth in general. Seems to make things more flexible 18:38:07 <nirik> on the openid front... we may want to look at mod_auth_openid. It looks like it might work to replace our mod_auth_pg thing if we wanted to. 18:38:07 <abadger1999> the one issue is that I don't believe we have an active member of our dev team that knows openid server-side 18:38:18 <abadger1999> so if it's broken in fas, we won't be able to fix it easily. 18:38:21 <skvidal> abadger1999: good point 18:38:38 * mdomsch is making _slow_ progress on MM 1.4. Bug fixes and perf improvements in the last couple weeks. I need to do more testing before I unleash it on the world - well after F17 launch 18:38:41 <abadger1999> But we're already depending on it for several things without issue. 18:39:07 <abadger1999> so... more doesn't hurt a whole lot and it does give other sorts of flexibility. 18:39:34 <jds2001> abadger1999: you can learn in 2 hours, right? :) 18:39:55 <abadger1999> jds2001: Yeah. two hours on the beach in the bahamas, you foot the bill ;-) 18:40:09 <pingou> sounds like a plan 18:40:12 <nirik> one advantage of mod_auth_openid is that you can run a script after the openid auth to determine if the auth is allowed or not, we could possibly hook that into the 2 factor stuff. 18:40:26 <abadger1999> nirik: Mmm that sounds nice. 18:40:38 * marcdeop joinns that plan 18:40:51 <nirik> http://findingscience.com/mod_auth_openid/ if anyone wants to look it over. 18:41:03 <nirik> it's under review (stalled), so we would need to finish that before we use it. 18:41:41 <nirik> anyhow, any more app news? or shall we move on? 18:41:58 <relrod> nothing else from me 18:42:08 <nirik> ok, upcoming items: 18:42:16 <nirik> #topic Upcoming Tasks/Items 18:42:16 <nirik> #info 2012-05-08 to 2012-05-29 FINAL FREEZE 18:42:16 <nirik> #info 2012-05-29 - F17 release 18:42:16 <nirik> #info 2012-06-01 - nag fi-apprentices. 18:42:16 <nirik> #info 2011-06-03 - gitweb-cache removal day. 18:42:16 <nirik> #info 2012-06-04 - class B reboots? 18:42:17 <nirik> #info 2012-06-05 - class A reboots? 18:42:19 <nirik> #info 2012-06-08 OOW: osuosl01.fedoraproject.org 18:42:20 <nirik> #info 2012-06-17 OOW: sign-vault02.phx2.fedoraproject.org 18:42:22 <nirik> #info 2012-06-21 to 2012-07-04 Kevin is off on trains and boats. 18:42:24 <nirik> I was looking at scheduling some mass rebooting for after freeze. 18:42:30 <nirik> get us up on the current kernel and updates. 18:43:11 <relrod> nirik: Historically the final freeze goes until one day after the release, not day-of 18:43:21 <nirik> yeah, sorry you are right. 18:43:45 * nirik adjusts 18:43:53 <nirik> anyhow, anything folks want to schedule or note? 18:44:26 <smooge> trip to Az in July is the only thing on my radar. 18:44:29 <inode0> voting will be happening June 1-7 18:44:44 <smooge> #info sometime in June hardware will show up in Phx for new systems 18:44:47 <dgilmore> nirik: going to look at doing some koji changes soon 18:44:49 <nirik> there's also town hall meetings coming up for the elections... everyone should go and ask hard questions. ;) 18:45:11 <nirik> smooge: cool. yeah, we should coordinate that with rhit folks and get it scheduled. 18:45:16 <nirik> dgilmore: whats on the slate? 18:45:23 * inode0 is hoping all the rebooting doesn't affect voting as we needed to extend once before for interruptions 18:45:45 <nirik> inode0: I could push them off a week... 18:45:48 <dgilmore> nirik: move to mod_wisgi 18:45:49 <nirik> and avoid that. 18:45:58 <dgilmore> wsgi 18:46:04 <smooge> mediawiki upgrade 18:46:24 <dgilmore> nirik: there has been somebiggish changes in koji 18:46:29 <dgilmore> upstream 18:46:35 <nirik> inode0: I'll do that. Thanks for the note. 18:46:53 <dgilmore> going to look at setting up some policies to limit certain builds to certain builders 18:46:53 <nirik> smooge: yeah, we need to get staging all updated so we can test... but would be good to get that done too. 18:47:18 <nirik> dgilmore: cool. Should we schedule that for when we do reboots? or will it be further down the road? 18:47:33 <dgilmore> nirik: maybe further down the road 18:47:38 <nirik> ok 18:47:52 <dgilmore> koji with mod_wsgi has not had a lot of testing yet 18:48:08 <dgilmore> its a pretty big change from mod_python 18:48:15 <nirik> yeah, but mod_python is dead. 18:48:24 <dgilmore> yerp 18:48:42 <lmacken> and we should secure our apps from hash collission attacks soon :) 18:49:01 <lmacken> **collision 18:49:10 <nirik> lmacken: yeah. Possibly we could do that at the same time as a reboot outage... 18:49:16 <nirik> although it shouldn't be much downtime 18:49:35 <lmacken> cool. 18:49:51 <nirik> if it's ready by then I guess. ;) 18:50:22 <lmacken> we should probably just take the environment variable approach first, since mod_wsgi upstream has been far from useful with regard to this patch 18:50:34 <nirik> yeah. seems easiest. 18:51:02 <nirik> #info mediawiki 119 upgrade coming. 18:51:09 <nirik> #info koji upgrades coming 18:51:25 <nirik> #info will apply fix for mod_wsgi hash stuff soon 18:51:41 <nirik> #info trip out onsite to phx2 sometime in the next few months. 18:51:47 <nirik> Anything else? 18:51:59 <mdomsch> MM 1.4 when I get around to it 18:52:23 <nirik> #info mirrormanager 1.4 coming too 18:52:40 * nirik wonders if we can get all this done before f18 freezes start. ;) 18:52:45 <nirik> time will tell 18:52:56 <nirik> #topic Open Floor 18:53:02 <mdomsch> S3 mirrors 18:53:03 * marcdeop encourages positive attitude! 18:53:07 <nirik> ok, anything for open floor? questions comments? 18:53:13 <mdomsch> I've started a new bucket for us-west-1 18:53:15 <nirik> mdomsch: oh yeah, whats the status there? 18:53:35 <mdomsch> and bapp02 is doing an s3cmd sync from s3://us-east-1 to s3://us-west-1 18:53:47 <mdomsch> so in reality s3 is doing the copy for us 18:53:50 <nirik> nice 18:53:56 <mdomsch> well... S3 is slow... 18:54:03 <mdomsch> it's been running for 3-4 days now 18:54:06 <mdomsch> since Monday 18:54:06 <nirik> how many regions should we/can we support? 18:54:23 <mdomsch> we could do several, but the copying is _really_ slow 18:54:47 <mdomsch> and arguably we should move archive (f8) content into its own set of buckets, so s3cmd list doesn't have to walk that content 18:54:58 <mdomsch> on every sync run 18:55:27 <nirik> yeah. 18:55:33 <mdomsch> I presume we prefer to let s3 do the copying (initial copy costs the OpenShift team about $11) 18:55:43 <mdomsch> into each bucket, rather than us push from phx2 into each bucket 18:55:47 <nirik> is this something we want to announce/shout out? either as part of release or seperate? 18:55:50 <mdomsch> so it becomes a 2-stage sync 18:56:01 <mdomsch> I don't know if it'll be done in time for the release... 18:56:08 <nirik> yeah, I'd say let them do it unless it's just too slow to work 18:56:16 <mdomsch> phx2 -> us-east-1; us-east-1->us-west-1 18:56:32 <mdomsch> it's taking days to move 300GB 18:56:44 <dgilmore> mdomsch: id say phx2 to us-west-1 18:56:46 <mdomsch> but I forget how long the initial upload from phx2 -> us-east-1 was 18:56:51 <dgilmore> just because its physically closer 18:57:05 <nirik> do we have any amazon contacts we could ping and ask? 18:57:16 * nirik looks at spevack. 18:57:16 <dgilmore> nirik: spevack 18:57:43 <mdomsch> anyhow, I can bring that online whenever; it's just a DNS change once the content is in place 18:57:52 <mdomsch> and a s3sync change to copy to a second destination 18:58:07 <nirik> mdomsch: can you ping spevack and see if he can provide any suggestions on how we can better do it? 18:58:17 * nirik can if you like too 18:58:22 <dgilmore> mdomsch: do we now what kinda hits the ec2 mirror gets? 18:58:37 <mdomsch> dgilmore: we have the logs; I don't have a log parsing tool to count them 18:58:51 <dgilmore> mdomsch: ok, mostly just curious 18:59:14 <mdomsch> many many many small files with 1-50 hits in each file 18:59:52 <mdomsch> that's all I have 18:59:55 <dgilmore> mdomsch: ok, mostly im curious as there has been zero feedback on the AMIS for f17 19:00:05 <dgilmore> makes me wonder if they have even been used 19:00:33 <marcdeop> amis? 19:00:46 <dgilmore> marcdeop: the fedora images in EC2 19:00:48 <nirik> amazon images 19:00:58 <skvidal> marcdeop: Amazon Machine Images 19:01:02 <skvidal> https://aws.amazon.com/amis 19:01:03 <nirik> so someone can say: "make me a new instance... Fedora 17 please" 19:01:04 <nirik> ;) 19:01:10 <marcdeop> ohh yeah thanks 19:01:22 <cyberworm54> :) 19:01:30 <cyberworm54> EC@ 19:01:47 <nirik> ok, any other open floor items? shall we call it a meeting? 19:02:34 <nirik> ok, thanks for coming everyone! 19:02:37 <dgilmore> nirik: think we are good 19:02:48 <nirik> as usual, hang out in #fedora-admin, #fedora-noc and #fedora-apps. 19:02:51 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
