16:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting, who's here 16:01 -!- maploin [n=oin@xxxxxxxxxxxxx] has joined #fedora-meeting 16:01 * jima , barely 16:01 * skvidal is 16:01 < mmcgrath> abadger1999: dgilmore f13 ianweller ivazquez lmacken marek mdomsch paulobanon skvidal: ping 16:02 < ivazquez> Pong. 16:02 < mmcgrath> ricky: ? 16:02 * abadger1999 here 16:02 * lmacken is around 16:03 < mmcgrath> aaalrighty. 16:03 < mmcgrath> lets get started. 16:03 < mmcgrath> this should be _real_ quick since we're in a change freeze. 16:03 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- F9 Release. 16:03 < mdomsch> yep 16:04 < ianweller> hi 16:04 < mmcgrath> .tiny https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&milestone=Fedora+9 16:04 < zodbot> mmcgrath: http://tinyurl.com/25vzyu 16:04 < mmcgrath> So thats what we've got left to do for the F9 launch 16:04 < marek> hey all 16:04 < mmcgrath> skvidal: is it your opinion we should wait until after F9 to switch to torrent.fedoraproject.org? 16:04 * ianweller just got home :D 16:04 < mmcgrath> marek: howdy 16:05 < skvidal> mmcgrath: at this point, yes 16:05 * mmcgrath agrees. 16:05 < skvidal> mmcgrath: even if we get the new ips - we've got no way of knowing if that actually solves it until more tests :-/ 16:05 < mmcgrath> it might make it now, but I have no reason to believe that the current torrent box will flake out in the next two weeks :) 16:05 < mmcgrath> .ticket 231 16:05 < zodbot> mmcgrath: #231 (post master directory space usage for mirrors) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/231 16:05 < mmcgrath> mdomsch: were you happy with the outcome on that enough that I can close the ticket? 16:06 < mdomsch> mmcgrath, yes 16:06 < mdomsch> mmcgrath, does that get run occasionally? 16:06 < mdomsch> e.g. cronjob? 16:06 < mmcgrath> mdomsch: yeah its a cron job. 16:06 * mmcgrath verifies its actually running 16:06 < mdomsch> perfect 16:07 < mmcgrath> mdomsch: it is - 30 19 * * * 16:07 < mdomsch> great 16:07 < mmcgrath> .any yingbull 16:07 < zodbot> mmcgrath: yingbull was last seen in #fedora-meeting 4 hours, 11 minutes, and 21 seconds ago: *** yingbull has quit IRC ("ChatZilla 0.9.78.1 [Firefox 2.0.0.13/2008031114]") 16:07 < mmcgrath> f13: you around? 16:08 < mmcgrath> we'll go back to that. 16:08 < mmcgrath> .ticket 333 16:08 < zodbot> mmcgrath: #333 (Add spam headers to bastion (smtp)) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/333 16:08 < mmcgrath> .ticket 54 16:08 < zodbot> mmcgrath: #54 (Postfix Server) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/54 16:08 -!- bzbot is now known as buggbot 16:08 < mmcgrath> those configs are ready to be deployed but there were concerns on the list about deploying this close to the release. 16:08 -!- hughsie [n=hughsie@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has left #fedora-meeting ["Ex-Chat"] 16:08 < mmcgrath> So shortly after the release we'll deploy. 16:08 < mmcgrath> .272 can be closed 16:09 < mmcgrath> .any susmit 16:09 < zodbot> mmcgrath: susmit was last seen in #fedora-meeting 1 week, 5 days, 7 hours, 41 minutes, and 11 seconds ago: *** susmit has parted #fedora-meeting () 16:09 -!- smooge [n=smooge@xxxxxxxxxxxxxxx] has quit "-ENOCAFFEINE" 16:09 < mmcgrath> I'm not sure whats going on with 16:09 < mmcgrath> .389 16:09 < mmcgrath> we haven't had an issue in a while but it'd be nice to have that shortly after F9 16:09 < mmcgrath> .ticket 389 16:09 < zodbot> mmcgrath: #389 (Monitor primary mirror) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/389 16:09 -!- petreu [n=peter@fedora/Standby] has quit "( www.nnscript.de :: NoNameScript 4.02 :: www.XLhost.de )" 16:10 < mmcgrath> So really there's nothing left to discuss on the F9 release. 16:10 < mmcgrath> One thing I'm looking at doing is finding the cheapest operation for our common pages via load tests here in my lab at home. 16:10 < mmcgrath> it turns out there's some fun stuff that goes on with mod_cache and mod_negotiation (which we use for translated websites) 16:10 < mmcgrath> there's a meeting at the end of the week to determine exactly what links we'll be giving out. 16:11 < mmcgrath> I had planned on setting the proxy servers to auto-cache just those pages in memory. 16:11 < mmcgrath> and their requisites (images, css, etc) 16:11 -!- sharkcz [n=dan@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 16:11 < mmcgrath> Anyone have any questions or concerns about the F9 release? 16:12 < mmcgrath> I'll take that as a no. 16:12 < mdomsch> mmcgrath, keeping the wiki online? 16:12 < ianweller> heh 16:12 < mdomsch> we won't have the new mediawiki up yet 16:13 < mmcgrath> mdomsch: yeah I'll be getting that list of pages during the meeting this week and creating static pages for the links we give out. 16:13 < mdomsch> even if it's r/o for release day 16:13 < mdomsch> ok 16:13 < mmcgrath> also, in theory, for this release we can add more readers. But I think creating static pages of our official links is probably the best strategy. 16:13 < ianweller> +1 16:13 < mdomsch> agreed 16:14 < mdomsch> does it make sense to add readers, and stop all writers for the day? 16:14 < mdomsch> or is that too hard? 16:14 < marek> I think we should leave few core writers to be able to edit the wiki, just in case of wrong info or typos 16:14 < mmcgrath> shouldn't be too hard. 16:15 < mmcgrath> marek: I'm actually more inclined to make them come to us first. In the past when the wiki has gotten "busy", writing makes it infinitly worse for some reason. 16:15 < mmcgrath> al though, in theory, we have the wiki setup in a way we never have before (and didn't have last time) 16:15 < mmcgrath> so ew can dedicate a writer and create a few readers. 16:15 -!- GeroldKa [n=GeroldKa@fedora/geroldka] has joined #fedora-meeting 16:16 < mmcgrath> I'll run a couple of tests and send a note to the list. 16:16 < mmcgrath> Ok, anyone else have any questions? 16:16 < mdomsch> mmcgrath, you mean we rewrite the edit URLs and point them at the same box? 16:16 < mdomsch> i'm good 16:16 < mmcgrath> mdomsch: well, right now all reads are going to app1, all writes are going to app2. We can make all reads go to app1,3,4 and leave all reads alone. 16:17 < mmcgrath> which isn't bad because app2 is a pretty busy box compared to the rest. 16:17 < mdomsch> nice 16:17 < mmcgrath> err "and leave all writes alone" 16:17 < mmcgrath> ok, we'll scoot along for now. 16:17 < ianweller> so we have separate read and write servers for moin? 16:17 < mmcgrath> ianweller: correct. 16:17 < ianweller> hmm. 16:17 < ianweller> ok! 16:17 < mmcgrath> ianweller: which is the most stable we've ever had it :) 16:17 < ianweller> tee hee 16:18 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Outage last week. 16:18 < mmcgrath> So we had two interesting outages this last week. 16:18 < mmcgrath> one of them, came and went on its own. DB2's load went way up, then came back down. 16:18 < mmcgrath> but the one before that, on a Friday or Sat. evening was very strange. 16:18 < mmcgrath> basically xen6 lost access to its storage subsystem. 16:18 < mmcgrath> I haven't had time to go through the logs yet but its not something we've seen before but is something we need to keep a closer eye on. 16:19 < mmcgrath> the good news is that because of the work susmit and I have been up to, our nagios false positives are way down. 16:19 -!- couf [n=bart@fedora/couf] has quit "leaving" 16:19 < mmcgrath> And the last thing on our agenda this week... 16:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Outage this Sat. 16:19 < mmcgrath> So PHX is going to be offline. 16:20 < skvidal> I'm sure nothing will break 16:20 < mmcgrath> They said they'll keep the outage to a minimum, but still, unlike nfs, we can't have our iscsi shares going up and down for 5 hours. 16:20 < mmcgrath> so I think we should just shut them all down. 16:20 < mmcgrath> So the basic plan is this. 16:20 < marek> I should be back at home and online to report the progress from network team 16:21 < mmcgrath> marek: thanks, that will be most helpful. 16:21 < mmcgrath> marek: do you know if they'll be switching out our switch? 16:21 < marek> I'm in Paris now and I hope I'll get back home till Friday 16:21 < mmcgrath> its EOL AFAIK. 16:21 < marek> mmcgrath: don't know, good question 16:21 < marek> mmcgrath: let me ask quick 16:22 < mmcgrath> marek: thanks. 16:22 < mmcgrath> So we'll be running all traffic through tummy.com for that outage. 16:22 < mmcgrath> we'll setup proxy3 to balance mirrormanager between app5 and app6. 16:23 < mmcgrath> since app2 is on local storage, it'll be going up and down as the network goes up and down, with no real harm. 16:23 < mmcgrath> so, with the exception of db2 (and therefore auth) the wiki at least and smolt will stay up when the network is available for them to do s. 16:23 < mmcgrath> unfortunately everything else will just be down. 16:24 < mdomsch> can tummy handle that? 16:24 < mdomsch> nirik was concerned that the secondary content rsync was eating their bandwidth 16:24 < mmcgrath> mdomsch: yeah, tummy handled all of our traffic during the F8 launch for a few hours. 16:24 -!- SmootherFrOgZ [n=Smoother@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Remote closed the connection 16:24 < mdomsch> ok, as long as they're ok with it 16:25 < nirik> yeah, that transfer was coming in on the much slower of our links for some reason... 16:25 < nirik> and it was for a long time. ;) 16:25 < mmcgrath> I don't think they'll even notice. I'll try to get an actual estimate of bw and talk to them ahead of time and see if we need to throttle or something. 16:25 < mmcgrath> nirik: right now though, can you see how much traffic is going through proxy3? It'll double. 16:25 * nirik goes to look 16:25 < mmcgrath> .dns fedoraproject.org 16:25 < zodbot> mmcgrath: 66.35.62.162 16:26 < nirik> Current Traffic: 117.3KiB/sec in/165.1KiB/sec out 16:26 < nirik> no biggie at all 16:26 < mmcgrath> so if we shutdown PHX and sent all traffic through you, that number would double. 16:26 < mmcgrath> yeah, thats what I thought. Good to verify it though. 16:26 < mmcgrath> Ok, Also during that outage, I'm going to reboot xen1 and give it the new 5.2 kernel. 16:27 < mmcgrath> we never did that during the last outage because the branching took much longer then we thought. 16:27 < mmcgrath> Anyone else have any concerns for this weekend? 16:27 < nirik> I don't think that would be any problem... 16:27 < marek> mmcgrath: ah, mgalgoci is out for training, I'll try to find more details tomorrow and let you know 16:28 < mdomsch> marek, see if malgoci got his internet2 netblock for download1 pls 16:29 < mdomsch> that'd be nice to have, but not critical 16:29 < skvidal> hey 16:29 < marek> mdomsch: afaik there's download-i2.fedora.redhat.com for that no? 16:29 < skvidal> mmcgrath just dropped off the net 16:29 < skvidal> he called me 16:29 < mdomsch> marek, don't know 16:29 < skvidal> anything else on the meeting? 16:30 < marek> mdomsch: from what I remember from mails, download1.fedora -> download3.fedora so people can download normalny 16:30 < marek> and download-i2.fedora.redhat.com is the inet2 16:30 < marek> skvidal: not for me 16:30 < mdomsch> marek, you've been out of the loop for a bit... 16:30 < mmcgrath> oh! I'm back. 16:30 < skvidal> marek: and ibiblio is getting things through th i2 connection 16:30 < skvidal> oh look 16:30 < skvidal> we have an mmcgrath again 16:30 < mmcgrath> marek: thanks. 16:30 < mdomsch> marek, I'll fill you in after the meeting 16:30 < marek> mdomsch: that's passible, passing my days in paris now :/ 16:31 < marek> k 16:31 < mmcgrath> Lets open the floor and close the meeting :) 16:31 < mdomsch> mmcgrath, good to prove our strategy for losing a data center 16:31 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor 16:32 < mdomsch> just before a release 16:32 < mmcgrath> Does anyone have anything else to discuss? 16:32 < mmcgrath> mdomsch: no doubt :) I hope for the next release we'll have the alternate vpn server up so the changes are automatic. 16:32 < abadger1999> skvidal: Are you holding off on the package-owners@xxxx aliases until after release? 16:32 < skvidal> abadger1999: yah - I'll tie it to postfix 16:33 < skvidal> abadger1999: b/c it'll be less effort (sorry, lazy) 16:34 < mmcgrath> skvidal: its also "the smart thing" to do :) 16:34 < skvidal> laziness frequently is 16:34 < skvidal> :) 16:34 < abadger1999> :-) 16:34 < mmcgrath> hah 16:34 < mmcgrath> Alrighty, anyone have anything else to discuss? 16:34 < mmcgrath> If not I'll close the meeting in 30 16:34 -!- rdieter is now known as rdieter_away 16:34 < mmcgrath> 10 16:35 < mmcgrath> 5 16:35 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting end!
Attachment:
pgpTyMd40KL7p.pgp
Description: PGP signature
_______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
