Meeting Log - 2007-10-04

[Ricky Zhou <ricky@xxxxxxxxxxxxxxxxx>]

16:00:06  * jima here
16:00:25  * mmcgrath looks for the two to drop
16:00:36 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:00:38 < mmcgrath> Who's here?
16:00:42  * lmacken 
16:00:42 -!- warren [i=warren@redhat/wombat/warren] has quit Remote closed the connection
16:00:42 -!- jeremy [i=katzj@nat/redhat/x-824cfb21e0d420e3] has quit Remote closed the connection
16:00:46 < jima> ha!
16:00:53 < ivazquez> Pong.
16:01:22 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has joined #fedora-meeting
16:01:23 < mmcgrath> dgilmore: paulobanon abadger1999 mbacovsk f13 skvidal jima + anyone I've forgotten
16:01:27 < mmcgrath> ping
16:01:36 < skvidal> pong
16:01:37 < frankc> Frank is eavesdropping
16:01:40 < abadger1999> pong
16:01:41 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has joined #fedora-meeting
16:01:41 < skvidal> and there go warren and jeremy
16:01:44 -!- |DrJef| [n=onefjef@fedora/Jef] has joined #fedora-meeting
16:01:47 < mmcgrath> :)
16:02:09 < warren> mmcgrath, damn you!
16:02:15 < mmcgrath> I didn't do it I swear!!!
16:02:17  * jima refrains from commentary on frank :)
16:02:22 < mmcgrath> heh
16:02:29 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
16:02:33 < mmcgrath> .tiny 
https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority
16:02:33 < mmcgrathbot> mmcgrath: http://tinyurl.com/yth34b
16:02:37 < jima> mmcgrath: could it be the topic change?
16:02:39 < warren> mmcgrath, do you change the topic or something?
16:02:41 < warren> I should put it under gdb...
16:02:52 -!- skvidal changed the topic of #fedora-meeting to: Infrastructure -- Tickets test
16:02:53 < mmcgrath> warren: it could be, after the meeting I'll change it again and see what happens.
16:03:00 < skvidal> nope, they didn't die
16:03:01 < warren> mmcgrath, k
16:03:03 < jima> warren: it was immediately after the topic change
16:03:08 < lmacken> lets start the meeting over and see if it happens again ;P
16:03:18 -!- jima changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:03:19 < warren> waiit....I don't have debuginfo
16:03:26 < mmcgrath> ok, first ticket relates to DNS -
16:03:30 < lmacken> (but not really)
16:03:33 < mmcgrath> #154
16:03:48 < mmcgrath> Jima spent some time this afternoon to create publictest[1-9].fedoraproject.org
16:03:51 < jima> bizarre.
16:03:58 < jima> mmcgrath: yeah, like 30 seconds. ;)
16:04:18 < lmacken> cool
16:04:27 < mmcgrath> This is mostly just so we can A) be consistent with the rest of our services and so 
                     we don't have to run DNS requests through RHIS.
16:04:42 < mmcgrath> I'll be setting up a RHEL box soon (I was waiting for this part to be done)
16:04:42 < jima> would anyone be too mad if i cleaned up the fp.o zone file? like, alphabetized it? or 
                 would that be too cvs-noisy?
16:04:58 < lmacken> is bastion the only one left to have .fedora.redhat.com ?
16:05:01 < mmcgrath> jima: actually I'd appreciate that, keep the 'balanced' pieces in their own section.
16:05:26 < mmcgrath> lmacken: I think so, there's a few things out there, like cvs.fedora.redhat.com 
                     still exists but so does cvs.fedoraproject.org
16:05:33 < jima> mmcgrath: that's fine, but "The rest of our configs"?
16:05:34 < mmcgrath> jima: while you're in there add a bastion.fedoraproject.org
16:05:58 < skvidal> jima: umm - but one thing
16:06:03 < skvidal> jima: test it, A LOT
16:06:05 < skvidal> before you push it
16:06:07 < jima> sections are fine, all the random hostnames are just annoying :)
16:06:15 < skvidal> b/c clipping dns is a really good way to cause a lot of noise
16:06:17 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
16:06:18 < mmcgrath> jima: I agree.
16:06:21 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:06:39 < jima> skvidal: enh...dns administration is a big part of my job.
16:06:43 < skvidal> cool
16:06:45 < skvidal> just making sure
16:06:57 < jima> i haven't made a dns boo-boo in YEARS :)
16:07:04  * nirik notes that named-checkzone is a handy command. ;) 
16:07:13 < jima> (that was 1998, when i put in a #comment in a zone file. yes, seriously.)
16:07:15 < skvidal> jima: no time like the present to start :)
16:07:23 < mmcgrath> <nod>
16:07:42 < mmcgrath> jima: so yeah I guess the config will have two sections both alphabetized.
16:07:44 < jima> *sigh* if only named-checkzone were on puppet1 :)
16:08:31 < mmcgrath> jima: you've got a workstation :-P
16:08:35 < nirik> it's in the main bind package along with named-checkconf (which is also very handy)
16:09:01 < mmcgrath> Anyone have any other questions regarding the DNS stuff thats going on?  I doubt 
                     that we'll just cut over to using it but we'll probably grandfather it in.
16:09:07 < jima> mmcgrath: yeah, i know -- i just do all my config editing on puppet1 :)
16:09:31  * jima will test this zone before pushing it
16:09:40 -!- stickster_work is now known as stickster_afk
16:09:48 < mmcgrath> ok, next ticket #170
16:09:58 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/170
16:10:00 < mmcgrathbot> mmcgrath: #170 (Hosting respins) - Fedora Infrastructure - Trac
16:10:01 -!- tibbs [i=tibbs@fedora/tibbs] has quit "Konversation terminated!"
16:10:18 < mmcgrath> I've been trying to find sponsorship for a new torrent server.
16:10:26 < mmcgrath> In the meantime I'd like us to create spins.fedoraproject.org
16:10:31 < skvidal> with an assload of disk space?
16:10:50 < mmcgrath> skvidal: I'm requesting 500G-1T of space + average throughput of 100Mb
16:10:57 < mmcgrath> err maximum throughput of 100Mb.
16:10:57 < skvidal> good
16:11:14 < mmcgrath> The problem I've been running into lately is getting the presentation layer of this 
                     stuff going.
16:11:25 < mmcgrath> I've hit up the art people and the websites list and we just don't have many actual 
                     designers out there.
16:11:53 < mmcgrath> mizmo has always come through for us in the past but I'd like to find another 
                     avenue, mostly because she's busy and because I know there's just got to be other 
                     people interested in getting involved in this stuff.
16:12:19 < mmcgrath> Anyone have any ideas on this?
16:12:38 < mmcgrath> I got 2 responses from the art team and followed up a few times with them but have 
                     yet to hear back about any proof of concepts or anything like that.
16:12:44 < ivazquez> I can help. Let me hit the -websites archive to see what's there.
16:13:21 < mmcgrath> ivazquez: thanks, we're actually moving to a templated based system with genshi 
                     soon so that should help.  I'm just surprised at how little interst there is in 
                     this stuff.  I'd think web designers would be all over it.
16:13:39  * ivazquez needs to learn Genshi anyways
16:14:02 < mmcgrath> thats really all I've got on #170.  its an F8 milestone and available to anyone 
                     who'd be interested in doing the work to set it up.
16:14:03  * jima sucks at design, like many sysadmins.
16:14:09  * mmcgrath does too.
16:14:19 < mmcgrath> jima: knowing you suck is half the battle :)
16:14:23 < mmcgrath> well, for us its the whole battle
16:14:37 < mmcgrath> which reminds me
16:14:38 < mmcgrath> f13: ping?
16:14:42 < mmcgrath> anyone from rel-eng ping?
16:14:49  * notting can play rel-eng on irc
16:15:13 < mmcgrath> notting: how far along has the gaming spin gotten?
16:15:19 < mmcgrath> as far as being blessed by releng?
16:15:22 < notting> iirc, jeremy was doing test spins
16:15:29 < mmcgrath> jeremy: ping?
16:15:44 < jeremy> I gave it a try yesterday ... had to fix up a few things, but nothing too bad
16:15:46  * mmcgrath just wants to make sure to use the gaming spin as a sort of template for how future 
          spins should go.
16:15:56 < notting> jeremy: so, it's doing the desktop stuff right now?
16:15:57 < mmcgrath> jeremy: do you suspect it will be ready prior to F8?
16:16:01 < jeremy> notting: yes
16:16:05 -!- cyberpear [n=cyberpea@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:16:09 < jeremy> mmcgrath: yep.  and potentially an xfce one too
16:16:16 < mmcgrath> excellent news.
16:16:42 < mmcgrath> so when those are ready ready I'll probably have to sit down with someone and get 
                     the workflow figured out.  I don't think there's anything too fancy going on there.
16:17:13 < mmcgrath> Ok, anyone have any questions regarding the respins?
16:17:20  * warren now has debuginfo for dircproxy!
16:17:22 < warren> restarting...
16:17:50 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has quit "Terminated with extreme prejudice 
          - dircproxy 1.2.0"
16:17:53 < mmcgrath> ok, on to the schedule
16:17:58 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Schedule
16:18:04 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/Schedule
16:18:29 < mmcgrath> Corporate Sponsorship - Nothing new to report here.
16:18:49 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has joined #fedora-meeting
16:19:02 < mmcgrath> I've been in discussions with Peer1.com, we've had quite a few meetings now.  The 
                     last one was with their engineers so I'm hopeful to hear something soon.
16:19:09 < mmcgrath> either a yes or a no at least.
16:19:29 < mmcgrath> thats really it on that front.  We've got funding to ship a server to the new 
                     german colo but I doubt that will be ready for F8.
16:19:39 < mmcgrath> Any questions regarding Corporate sponsorship?  Any new leads?
16:19:51 < skvidal> mmcgrath: jeff_s
16:19:52 < mdomsch> mmcgrath, you have netapp on your list?
16:20:10 < mmcgrath> skvidal: will send an email soon
16:20:28 < mmcgrath> mdomsch: not yet, I think thats my fault, you gave me a contact over there I think 
                     but I never followed up.
16:20:49  * mmcgrath remembers now, I asked Chris Smith how much maintanence on our current netapp was 
          but never heard back.
16:21:17 < mdomsch> I spoke to Trond at LKS a few weeks ago, he seemed to think it was possible
16:21:33 < mmcgrath> mdomsch: I know you talked with one of the netapp guys, could you send me another 
                     email with all of his contact info?
16:21:39 < mdomsch> will do
16:21:46 < mmcgrath> much appreciated.
16:22:04 < mmcgrath> Anyone have anything else?
16:22:11 -!- paulobanon_ [n=psantos@xxxxxxxxxxxxxxx] has joined #fedora-meeting
16:22:23 < mmcgrath> Ok, moving on
16:22:23  * paulobanon_ is here now :)
16:22:25 < mmcgrath> paulobanon_: hey
16:22:32 < paulobanon_> back from our colo in DE :)
16:22:33 < mmcgrath> Nothing new on the architecutral documentation
16:22:45  * jima cheers paulobanon_ on
16:22:51 < mmcgrath> paulobanon_: the one thats also soon to be Fedora's colo in DE?
16:22:56 < paulobanon_> yup
16:23:08 < mmcgrath> solid
16:23:13 < mmcgrath> Nothing new in the SOP front
16:23:22 < mmcgrath> jima: did you apply for the syadmin-devel group?
16:23:33 < jima> mmcgrath: yeah, abadger1999 approved me
16:23:46 < mmcgrath> cool
16:23:55 < mmcgrath> so jima's got new sponsorship in the sysadmin-devel group.
16:24:13 < mmcgrath> Thats it for the schedule, I've got two other things I'd like to talk about before 
                     we open the floor.
16:24:21  * jima hides
16:24:26 < mmcgrath> one is another ticket.
16:24:44 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Misc Items
16:24:46  * warren OK, gdb is now watching this process.
16:24:49 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/93
16:24:51 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:25:11 < mmcgrath> warren: cool
16:25:15 < mmcgrath> So Fedora 8 is on the way
16:25:43 < mmcgrath> The current schedule says its going to be on November 8th.
16:25:55 < skvidal> mmcgrath: where will you be on nov 8th?
16:26:10 < mmcgrath> For right now that means that our Infrastructure freeze is on November 1st.
16:26:20 < mmcgrath> skvidal: I believe I'll still be here on November 8th but not on the 9th.
16:26:24 < skvidal> :)
16:26:37 < mmcgrath> So here's a roundup of where we are right this second.
16:26:55 < mmcgrath> We've got two proxy servers being load balanced in PHX each with 1G ram and each 
                     with 1 bad drive in it.
16:27:02 < mmcgrath> I've got a ticket in to replace each of those bad drives.
16:27:35 < mmcgrath> At present we have proxy3 in Denver with tummy.com.  Its got 5G ram dedicated to it.
16:27:49 < mmcgrath> And good drives, you get the idea.
16:27:50 < skvidal> odds on the disks being swapped out before f8?
16:27:59 < mmcgrath> skvidal: fair to good
16:28:08 < skvidal> mmcgrath: got an eta at all, yet?
16:28:10 < mmcgrath> its going to be one of stacy's final fairwells.
16:28:11 < mmcgrath> not yet.
16:28:14 < skvidal> ok
16:28:16 < skvidal> cool
16:28:20 < paulobanon_> oods on proxy4 before f8 ?
16:28:28 < paulobanon_> *odds
16:28:34 < mmcgrath> paulobanon_: not good, I just don't think we'll have one in before the 
                     infrastructure freeze.
16:28:43 < mmcgrath> having said that though, I don't think we'll *need* one.
16:28:46 < paulobanon_> oh well
16:29:08 < mmcgrath> I've also requested at the same time as the drive swaps happen that we make sure 
                     that some of the xen servers have a nic in the proxy network and the non-proxy 
                     network.
16:29:17 < mmcgrath> so we should be able to create proxy servers for the release in phx.
16:29:41 < mdomsch> and websites will keep the size of the front page small again, yes?
16:29:44 < jima> mmcgrath: i've done dual-nic setups with xen, yeah
16:29:46 < mmcgrath> I'm also thinking about adding an additional IP address to the tummy.com server so 
                     that twice as much traffic ends up there on release day.  My experience is that its 
                     just more peppy and with the RAM it will be better.
16:29:58 < mmcgrath> mdomsch: yes, probably very similar to what it is right now.
16:30:04 < mdomsch> goodness
16:30:19 < mmcgrath> so really, we could be in better shape, but we have a lot of options.
16:30:29 < mmcgrath> Something to note right now is that only fedoraproject.org email is going through 
                     proxy3.
16:30:43 < mmcgrath> not docs, not admin, not translate, not hosted, not koji, etc.
16:30:43 < skvidal> no other mx?
16:30:47 < skvidal> oh, I see
16:30:50 < mmcgrath> errrr sorry
16:30:53 < mmcgrath> s/email/web traffic/
16:30:56  * nirik is glad proxy3 is working out nicely. If you need anything on it, let me or jafo know. 
16:31:14 < jima> mmcgrath: you have no idea how confused you just made me :)
16:31:15 < mmcgrath> nirik: honestly it could very well save our butts this release so its going very 
                     well.  Thank you.
16:31:19 < mmcgrath> jima: sorry about that.
16:31:42 < mmcgrath> One thing I'd like to do is migrate docs over to fp.o.  That should be as simple as 
                     changing the CNAME and making sure that docs.fp.o is listening to "fp.o"
16:31:49 < mmcgrath> Those are our two major hitters.
16:32:00 < mmcgrath> Also this time around we have cachign enabled on the proxy servers.
16:32:19 < mmcgrath> We can alter it so that getfedora and the release notes are all served from memory.
16:32:27 < paulobanon_> yup
16:32:37 < mmcgrath> This stuff is all a little ways off but its good to start talking about it now.
16:32:48 < abadger1999> Thanks paulo!
16:32:53 < paulobanon_> and if any TG app wants it also, dont forget the /static/ :)
16:33:00 < mmcgrath> Does anyone else have any questions related to the F8 release?  I've added it to 
                     the meeting notes so we'll be discussing it from here on out.
16:33:10 < mmcgrath> paulobanon_: did we figure out the whole session issues with that?
16:33:38 < mmcgrath> paulobanon_: sorry we can talk about that later, probably not that important for 
                     this meeting.
16:33:46 < mmcgrath> The other thing I wanted to talk about is our mirrors setup
16:33:47 < mmcgrath> f13: ping
16:33:49 < mmcgrath> lmacken: ping?
16:34:00 < lmacken> mmcgrath: pong
16:34:08 < paulobanon_> mmcgrath: we can talk about that after the meeting in #fedora-admin
16:34:43 < mmcgrath> lmacken: so we've had a lot of problems with the mirrors over the last month, we've 
                     blown out updates 2 times.  the test3 release got copied to the wrong spot, test2 
                     is no longer available and the comps file that shipped last night was bad somehow.
16:34:54 < mmcgrath> I know some of that was our tools and some of that was a typo
16:34:59 < mmcgrath> and lastnights thing we just don't know about.
16:35:10 < jima> yeah :(
16:35:10 -!- Lovechild [n=david@about/unixlove/Lovechild] has quit "Ex-Chat"
16:35:37 < mmcgrath> is there anything that we can be doing right now to make sure this stuf doesn't 
                     happen anymore?  I know you're working on mash, will that fix all of these issues?
16:36:12 < mmcgrath> do we need to dedicate more resources to it?
16:36:13 < lmacken> well, we don't have much control with what IS pushes out.. but I think it couldn't 
                    hurt to add a bit more paranoia to our tools before they flip the live bits.
16:36:24 < paulobanon_> were all problems in the tools been identified ?
16:36:30 < lmacken> even with all of the sanity checking bodhi could have done, it wouldn't have stopped 
                    what happened today with the comps..
16:36:42 < jima> how do we push bits out? rsync?
16:36:51 < lmacken> the repo that bodhi mashed was fine, but it synced out with a bad comps
16:36:57 < mmcgrath> lmacken: can't we add a check to bodhi does the sha1sum match whats in the xml file?
16:37:09 < lmacken> I haven't worked on mash at all..
16:37:14 < mmcgrath> <nod>
16:37:20 < lmacken> mmcgrath: yeah.. but that wouldn't have helped today, since everything matched up 
                    fine
16:00:06  * jima here
16:00:25  * mmcgrath looks for the two to drop
16:00:36 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:00:38 < mmcgrath> Who's here?
16:00:42  * lmacken 
16:00:42 -!- warren [i=warren@redhat/wombat/warren] has quit Remote closed the connection
16:00:42 -!- jeremy [i=katzj@nat/redhat/x-824cfb21e0d420e3] has quit Remote closed the connection
16:00:46 < jima> ha!
16:00:53 < ivazquez> Pong.
16:01:22 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has joined #fedora-meeting
16:01:23 < mmcgrath> dgilmore: paulobanon abadger1999 mbacovsk f13 skvidal jima + anyone I've forgotten
16:01:27 < mmcgrath> ping
16:01:36 < skvidal> pong
16:01:37 < frankc> Frank is eavesdropping
16:01:40 < abadger1999> pong
16:01:41 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has joined #fedora-meeting
16:01:41 < skvidal> and there go warren and jeremy
16:01:44 -!- |DrJef| [n=onefjef@fedora/Jef] has joined #fedora-meeting
16:01:47 < mmcgrath> :)
16:02:09 < warren> mmcgrath, damn you!
16:02:15 < mmcgrath> I didn't do it I swear!!!
16:02:17  * jima refrains from commentary on frank :)
16:02:22 < mmcgrath> heh
16:02:29 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
16:02:33 < mmcgrath> .tiny 
https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority
16:02:33 < mmcgrathbot> mmcgrath: http://tinyurl.com/yth34b
16:02:37 < jima> mmcgrath: could it be the topic change?
16:02:39 < warren> mmcgrath, do you change the topic or something?
16:02:41 < warren> I should put it under gdb...
16:02:52 -!- skvidal changed the topic of #fedora-meeting to: Infrastructure -- Tickets test
16:02:53 < mmcgrath> warren: it could be, after the meeting I'll change it again and see what happens.
16:03:00 < skvidal> nope, they didn't die
16:03:01 < warren> mmcgrath, k
16:03:03 < jima> warren: it was immediately after the topic change
16:03:08 < lmacken> lets start the meeting over and see if it happens again ;P
16:03:18 -!- jima changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:03:19 < warren> waiit....I don't have debuginfo
16:03:26 < mmcgrath> ok, first ticket relates to DNS -
16:03:30 < lmacken> (but not really)
16:03:33 < mmcgrath> #154
16:03:48 < mmcgrath> Jima spent some time this afternoon to create publictest[1-9].fedoraproject.org
16:03:51 < jima> bizarre.
16:03:58 < jima> mmcgrath: yeah, like 30 seconds. ;)
16:04:18 < lmacken> cool
16:04:27 < mmcgrath> This is mostly just so we can A) be consistent with the rest of our services and so 
                     we don't have to run DNS requests through RHIS.
16:04:42 < mmcgrath> I'll be setting up a RHEL box soon (I was waiting for this part to be done)
16:04:42 < jima> would anyone be too mad if i cleaned up the fp.o zone file? like, alphabetized it? or 
                 would that be too cvs-noisy?
16:04:58 < lmacken> is bastion the only one left to have .fedora.redhat.com ?
16:05:01 < mmcgrath> jima: actually I'd appreciate that, keep the 'balanced' pieces in their own section.
16:05:26 < mmcgrath> lmacken: I think so, there's a few things out there, like cvs.fedora.redhat.com 
                     still exists but so does cvs.fedoraproject.org
16:05:33 < jima> mmcgrath: that's fine, but "The rest of our configs"?
16:05:34 < mmcgrath> jima: while you're in there add a bastion.fedoraproject.org
16:05:58 < skvidal> jima: umm - but one thing
16:06:03 < skvidal> jima: test it, A LOT
16:06:05 < skvidal> before you push it
16:06:07 < jima> sections are fine, all the random hostnames are just annoying :)
16:06:15 < skvidal> b/c clipping dns is a really good way to cause a lot of noise
16:06:17 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
16:06:18 < mmcgrath> jima: I agree.
16:06:21 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:06:39 < jima> skvidal: enh...dns administration is a big part of my job.
16:06:43 < skvidal> cool
16:06:45 < skvidal> just making sure
16:06:57 < jima> i haven't made a dns boo-boo in YEARS :)
16:07:04  * nirik notes that named-checkzone is a handy command. ;) 
16:07:13 < jima> (that was 1998, when i put in a #comment in a zone file. yes, seriously.)
16:07:15 < skvidal> jima: no time like the present to start :)
16:07:23 < mmcgrath> <nod>
16:07:42 < mmcgrath> jima: so yeah I guess the config will have two sections both alphabetized.
16:07:44 < jima> *sigh* if only named-checkzone were on puppet1 :)
16:08:31 < mmcgrath> jima: you've got a workstation :-P
16:08:35 < nirik> it's in the main bind package along with named-checkconf (which is also very handy)
16:09:01 < mmcgrath> Anyone have any other questions regarding the DNS stuff thats going on?  I doubt 
                     that we'll just cut over to using it but we'll probably grandfather it in.
16:09:07 < jima> mmcgrath: yeah, i know -- i just do all my config editing on puppet1 :)
16:09:31  * jima will test this zone before pushing it
16:09:40 -!- stickster_work is now known as stickster_afk
16:09:48 < mmcgrath> ok, next ticket #170
16:09:58 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/170
16:10:00 < mmcgrathbot> mmcgrath: #170 (Hosting respins) - Fedora Infrastructure - Trac
16:10:01 -!- tibbs [i=tibbs@fedora/tibbs] has quit "Konversation terminated!"
16:10:18 < mmcgrath> I've been trying to find sponsorship for a new torrent server.
16:10:26 < mmcgrath> In the meantime I'd like us to create spins.fedoraproject.org
16:10:31 < skvidal> with an assload of disk space?
16:10:50 < mmcgrath> skvidal: I'm requesting 500G-1T of space + average throughput of 100Mb
16:10:57 < mmcgrath> err maximum throughput of 100Mb.
16:10:57 < skvidal> good
16:11:14 < mmcgrath> The problem I've been running into lately is getting the presentation layer of this 
                     stuff going.
16:11:25 < mmcgrath> I've hit up the art people and the websites list and we just don't have many actual 
                     designers out there.
16:11:53 < mmcgrath> mizmo has always come through for us in the past but I'd like to find another 
                     avenue, mostly because she's busy and because I know there's just got to be other 
                     people interested in getting involved in this stuff.
16:12:19 < mmcgrath> Anyone have any ideas on this?
16:12:38 < mmcgrath> I got 2 responses from the art team and followed up a few times with them but have 
                     yet to hear back about any proof of concepts or anything like that.
16:12:44 < ivazquez> I can help. Let me hit the -websites archive to see what's there.
16:13:21 < mmcgrath> ivazquez: thanks, we're actually moving to a templated based system with genshi 
                     soon so that should help.  I'm just surprised at how little interst there is in 
                     this stuff.  I'd think web designers would be all over it.
16:13:39  * ivazquez needs to learn Genshi anyways
16:14:02 < mmcgrath> thats really all I've got on #170.  its an F8 milestone and available to anyone 
                     who'd be interested in doing the work to set it up.
16:14:03  * jima sucks at design, like many sysadmins.
16:14:09  * mmcgrath does too.
16:14:19 < mmcgrath> jima: knowing you suck is half the battle :)
16:14:23 < mmcgrath> well, for us its the whole battle
16:14:37 < mmcgrath> which reminds me
16:14:38 < mmcgrath> f13: ping?
16:14:42 < mmcgrath> anyone from rel-eng ping?
16:14:49  * notting can play rel-eng on irc
16:15:13 < mmcgrath> notting: how far along has the gaming spin gotten?
16:15:19 < mmcgrath> as far as being blessed by releng?
16:15:22 < notting> iirc, jeremy was doing test spins
16:15:29 < mmcgrath> jeremy: ping?
16:15:44 < jeremy> I gave it a try yesterday ... had to fix up a few things, but nothing too bad
16:15:46  * mmcgrath just wants to make sure to use the gaming spin as a sort of template for how future 
          spins should go.
16:15:56 < notting> jeremy: so, it's doing the desktop stuff right now?
16:15:57 < mmcgrath> jeremy: do you suspect it will be ready prior to F8?
16:16:01 < jeremy> notting: yes
16:16:05 -!- cyberpear [n=cyberpea@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:16:09 < jeremy> mmcgrath: yep.  and potentially an xfce one too
16:16:16 < mmcgrath> excellent news.
16:16:42 < mmcgrath> so when those are ready ready I'll probably have to sit down with someone and get 
                     the workflow figured out.  I don't think there's anything too fancy going on there.
16:17:13 < mmcgrath> Ok, anyone have any questions regarding the respins?
16:17:20  * warren now has debuginfo for dircproxy!
16:17:22 < warren> restarting...
16:17:50 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has quit "Terminated with extreme prejudice 
          - dircproxy 1.2.0"
16:17:53 < mmcgrath> ok, on to the schedule
16:17:58 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Schedule
16:18:04 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/Schedule
16:18:29 < mmcgrath> Corporate Sponsorship - Nothing new to report here.
16:18:49 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has joined #fedora-meeting
16:19:02 < mmcgrath> I've been in discussions with Peer1.com, we've had quite a few meetings now.  The 
                     last one was with their engineers so I'm hopeful to hear something soon.
16:19:09 < mmcgrath> either a yes or a no at least.
16:19:29 < mmcgrath> thats really it on that front.  We've got funding to ship a server to the new 
                     german colo but I doubt that will be ready for F8.
16:19:39 < mmcgrath> Any questions regarding Corporate sponsorship?  Any new leads?
16:19:51 < skvidal> mmcgrath: jeff_s
16:19:52 < mdomsch> mmcgrath, you have netapp on your list?
16:20:10 < mmcgrath> skvidal: will send an email soon
16:20:28 < mmcgrath> mdomsch: not yet, I think thats my fault, you gave me a contact over there I think 
                     but I never followed up.
16:20:49  * mmcgrath remembers now, I asked Chris Smith how much maintanence on our current netapp was 
          but never heard back.
16:21:17 < mdomsch> I spoke to Trond at LKS a few weeks ago, he seemed to think it was possible
16:21:33 < mmcgrath> mdomsch: I know you talked with one of the netapp guys, could you send me another 
                     email with all of his contact info?
16:21:39 < mdomsch> will do
16:21:46 < mmcgrath> much appreciated.
16:22:04 < mmcgrath> Anyone have anything else?
16:22:11 -!- paulobanon_ [n=psantos@xxxxxxxxxxxxxxx] has joined #fedora-meeting
16:22:23 < mmcgrath> Ok, moving on
16:22:23  * paulobanon_ is here now :)
16:22:25 < mmcgrath> paulobanon_: hey
16:22:32 < paulobanon_> back from our colo in DE :)
16:22:33 < mmcgrath> Nothing new on the architecutral documentation
16:22:45  * jima cheers paulobanon_ on
16:22:51 < mmcgrath> paulobanon_: the one thats also soon to be Fedora's colo in DE?
16:22:56 < paulobanon_> yup
16:23:08 < mmcgrath> solid
16:23:13 < mmcgrath> Nothing new in the SOP front
16:23:22 < mmcgrath> jima: did you apply for the syadmin-devel group?
16:23:33 < jima> mmcgrath: yeah, abadger1999 approved me
16:23:46 < mmcgrath> cool
16:23:55 < mmcgrath> so jima's got new sponsorship in the sysadmin-devel group.
16:24:13 < mmcgrath> Thats it for the schedule, I've got two other things I'd like to talk about before 
                     we open the floor.
16:24:21  * jima hides
16:24:26 < mmcgrath> one is another ticket.
16:24:44 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Misc Items
16:24:46  * warren OK, gdb is now watching this process.
16:24:49 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/93
16:24:51 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:25:11 < mmcgrath> warren: cool
16:25:15 < mmcgrath> So Fedora 8 is on the way
16:25:43 < mmcgrath> The current schedule says its going to be on November 8th.
16:25:55 < skvidal> mmcgrath: where will you be on nov 8th?
16:26:10 < mmcgrath> For right now that means that our Infrastructure freeze is on November 1st.
16:26:20 < mmcgrath> skvidal: I believe I'll still be here on November 8th but not on the 9th.
16:26:24 < skvidal> :)
16:26:37 < mmcgrath> So here's a roundup of where we are right this second.
16:26:55 < mmcgrath> We've got two proxy servers being load balanced in PHX each with 1G ram and each 
                     with 1 bad drive in it.
16:27:02 < mmcgrath> I've got a ticket in to replace each of those bad drives.
16:27:35 < mmcgrath> At present we have proxy3 in Denver with tummy.com.  Its got 5G ram dedicated to it.
16:27:49 < mmcgrath> And good drives, you get the idea.
16:27:50 < skvidal> odds on the disks being swapped out before f8?
16:27:59 < mmcgrath> skvidal: fair to good
16:28:08 < skvidal> mmcgrath: got an eta at all, yet?
16:28:10 < mmcgrath> its going to be one of stacy's final fairwells.
16:28:11 < mmcgrath> not yet.
16:28:14 < skvidal> ok
16:28:16 < skvidal> cool
16:28:20 < paulobanon_> oods on proxy4 before f8 ?
16:28:28 < paulobanon_> *odds
16:28:34 < mmcgrath> paulobanon_: not good, I just don't think we'll have one in before the 
                     infrastructure freeze.
16:28:43 < mmcgrath> having said that though, I don't think we'll *need* one.
16:28:46 < paulobanon_> oh well
16:29:08 < mmcgrath> I've also requested at the same time as the drive swaps happen that we make sure 
                     that some of the xen servers have a nic in the proxy network and the non-proxy 
                     network.
16:29:17 < mmcgrath> so we should be able to create proxy servers for the release in phx.
16:29:41 < mdomsch> and websites will keep the size of the front page small again, yes?
16:29:44 < jima> mmcgrath: i've done dual-nic setups with xen, yeah
16:29:46 < mmcgrath> I'm also thinking about adding an additional IP address to the tummy.com server so 
                     that twice as much traffic ends up there on release day.  My experience is that its 
                     just more peppy and with the RAM it will be better.
16:29:58 < mmcgrath> mdomsch: yes, probably very similar to what it is right now.
16:30:04 < mdomsch> goodness
16:30:19 < mmcgrath> so really, we could be in better shape, but we have a lot of options.
16:30:29 < mmcgrath> Something to note right now is that only fedoraproject.org email is going through 
                     proxy3.
16:30:43 < mmcgrath> not docs, not admin, not translate, not hosted, not koji, etc.
16:30:43 < skvidal> no other mx?
16:30:47 < skvidal> oh, I see
16:30:50 < mmcgrath> errrr sorry
16:30:53 < mmcgrath> s/email/web traffic/
16:30:56  * nirik is glad proxy3 is working out nicely. If you need anything on it, let me or jafo know. 
16:31:14 < jima> mmcgrath: you have no idea how confused you just made me :)
16:31:15 < mmcgrath> nirik: honestly it could very well save our butts this release so its going very 
                     well.  Thank you.
16:31:19 < mmcgrath> jima: sorry about that.
16:31:42 < mmcgrath> One thing I'd like to do is migrate docs over to fp.o.  That should be as simple as 
                     changing the CNAME and making sure that docs.fp.o is listening to "fp.o"
16:31:49 < mmcgrath> Those are our two major hitters.
16:32:00 < mmcgrath> Also this time around we have cachign enabled on the proxy servers.
16:32:19 < mmcgrath> We can alter it so that getfedora and the release notes are all served from memory.
16:32:27 < paulobanon_> yup
16:32:37 < mmcgrath> This stuff is all a little ways off but its good to start talking about it now.
16:32:48 < abadger1999> Thanks paulo!
16:32:53 < paulobanon_> and if any TG app wants it also, dont forget the /static/ :)
16:33:00 < mmcgrath> Does anyone else have any questions related to the F8 release?  I've added it to 
                     the meeting notes so we'll be discussing it from here on out.
16:33:10 < mmcgrath> paulobanon_: did we figure out the whole session issues with that?
16:33:38 < mmcgrath> paulobanon_: sorry we can talk about that later, probably not that important for 
                     this meeting.
16:33:46 < mmcgrath> The other thing I wanted to talk about is our mirrors setup
16:33:47 < mmcgrath> f13: ping
16:33:49 < mmcgrath> lmacken: ping?
16:34:00 < lmacken> mmcgrath: pong
16:34:08 < paulobanon_> mmcgrath: we can talk about that after the meeting in #fedora-admin
16:34:43 < mmcgrath> lmacken: so we've had a lot of problems with the mirrors over the last month, we've 
                     blown out updates 2 times.  the test3 release got copied to the wrong spot, test2 
                     is no longer available and the comps file that shipped last night was bad somehow.
16:34:54 < mmcgrath> I know some of that was our tools and some of that was a typo
16:34:59 < mmcgrath> and lastnights thing we just don't know about.
16:35:10 < jima> yeah :(
16:35:10 -!- Lovechild [n=david@about/unixlove/Lovechild] has quit "Ex-Chat"
16:35:37 < mmcgrath> is there anything that we can be doing right now to make sure this stuf doesn't 
                     happen anymore?  I know you're working on mash, will that fix all of these issues?
16:36:12 < mmcgrath> do we need to dedicate more resources to it?
16:36:13 < lmacken> well, we don't have much control with what IS pushes out.. but I think it couldn't 
                    hurt to add a bit more paranoia to our tools before they flip the live bits.
16:36:24 < paulobanon_> were all problems in the tools been identified ?
16:36:30 < lmacken> even with all of the sanity checking bodhi could have done, it wouldn't have stopped 
                    what happened today with the comps..
16:36:42 < jima> how do we push bits out? rsync?
16:36:51 < lmacken> the repo that bodhi mashed was fine, but it synced out with a bad comps
16:36:57 < mmcgrath> lmacken: can't we add a check to bodhi does the sha1sum match whats in the xml file?
16:37:09 < lmacken> I haven't worked on mash at all..
16:37:14 < mmcgrath> <nod>
16:37:20 < lmacken> mmcgrath: yeah.. but that wouldn't have helped today, since everything matched up 
                    fine
16:00:06  * jima here
16:00:25  * mmcgrath looks for the two to drop
16:00:36 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:00:38 < mmcgrath> Who's here?
16:00:42  * lmacken 
16:00:42 -!- warren [i=warren@redhat/wombat/warren] has quit Remote closed the connection
16:00:42 -!- jeremy [i=katzj@nat/redhat/x-824cfb21e0d420e3] has quit Remote closed the connection
16:00:46 < jima> ha!
16:00:53 < ivazquez> Pong.
16:01:22 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has joined #fedora-meeting
16:01:23 < mmcgrath> dgilmore: paulobanon abadger1999 mbacovsk f13 skvidal jima + anyone I've forgotten
16:01:27 < mmcgrath> ping
16:01:36 < skvidal> pong
16:01:37 < frankc> Frank is eavesdropping
16:01:40 < abadger1999> pong
16:01:41 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has joined #fedora-meeting
16:01:41 < skvidal> and there go warren and jeremy
16:01:44 -!- |DrJef| [n=onefjef@fedora/Jef] has joined #fedora-meeting
16:01:47 < mmcgrath> :)
16:02:09 < warren> mmcgrath, damn you!
16:02:15 < mmcgrath> I didn't do it I swear!!!
16:02:17  * jima refrains from commentary on frank :)
16:02:22 < mmcgrath> heh
16:02:29 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
16:02:33 < mmcgrath> .tiny 
https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority
16:02:33 < mmcgrathbot> mmcgrath: http://tinyurl.com/yth34b
16:02:37 < jima> mmcgrath: could it be the topic change?
16:02:39 < warren> mmcgrath, do you change the topic or something?
16:02:41 < warren> I should put it under gdb...
16:02:52 -!- skvidal changed the topic of #fedora-meeting to: Infrastructure -- Tickets test
16:02:53 < mmcgrath> warren: it could be, after the meeting I'll change it again and see what happens.
16:03:00 < skvidal> nope, they didn't die
16:03:01 < warren> mmcgrath, k
16:03:03 < jima> warren: it was immediately after the topic change
16:03:08 < lmacken> lets start the meeting over and see if it happens again ;P
16:03:18 -!- jima changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:03:19 < warren> waiit....I don't have debuginfo
16:03:26 < mmcgrath> ok, first ticket relates to DNS -
16:03:30 < lmacken> (but not really)
16:03:33 < mmcgrath> #154
16:03:48 < mmcgrath> Jima spent some time this afternoon to create publictest[1-9].fedoraproject.org
16:03:51 < jima> bizarre.
16:03:58 < jima> mmcgrath: yeah, like 30 seconds. ;)
16:04:18 < lmacken> cool
16:04:27 < mmcgrath> This is mostly just so we can A) be consistent with the rest of our services and so 
                     we don't have to run DNS requests through RHIS.
16:04:42 < mmcgrath> I'll be setting up a RHEL box soon (I was waiting for this part to be done)
16:04:42 < jima> would anyone be too mad if i cleaned up the fp.o zone file? like, alphabetized it? or 
                 would that be too cvs-noisy?
16:04:58 < lmacken> is bastion the only one left to have .fedora.redhat.com ?
16:05:01 < mmcgrath> jima: actually I'd appreciate that, keep the 'balanced' pieces in their own section.
16:05:26 < mmcgrath> lmacken: I think so, there's a few things out there, like cvs.fedora.redhat.com 
                     still exists but so does cvs.fedoraproject.org
16:05:33 < jima> mmcgrath: that's fine, but "The rest of our configs"?
16:05:34 < mmcgrath> jima: while you're in there add a bastion.fedoraproject.org
16:05:58 < skvidal> jima: umm - but one thing
16:06:03 < skvidal> jima: test it, A LOT
16:06:05 < skvidal> before you push it
16:06:07 < jima> sections are fine, all the random hostnames are just annoying :)
16:06:15 < skvidal> b/c clipping dns is a really good way to cause a lot of noise
16:06:17 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
16:06:18 < mmcgrath> jima: I agree.
16:06:21 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:06:39 < jima> skvidal: enh...dns administration is a big part of my job.
16:06:43 < skvidal> cool
16:06:45 < skvidal> just making sure
16:06:57 < jima> i haven't made a dns boo-boo in YEARS :)
16:07:04  * nirik notes that named-checkzone is a handy command. ;) 
16:07:13 < jima> (that was 1998, when i put in a #comment in a zone file. yes, seriously.)
16:07:15 < skvidal> jima: no time like the present to start :)
16:07:23 < mmcgrath> <nod>
16:07:42 < mmcgrath> jima: so yeah I guess the config will have two sections both alphabetized.
16:07:44 < jima> *sigh* if only named-checkzone were on puppet1 :)
16:08:31 < mmcgrath> jima: you've got a workstation :-P
16:08:35 < nirik> it's in the main bind package along with named-checkconf (which is also very handy)
16:09:01 < mmcgrath> Anyone have any other questions regarding the DNS stuff thats going on?  I doubt 
                     that we'll just cut over to using it but we'll probably grandfather it in.
16:09:07 < jima> mmcgrath: yeah, i know -- i just do all my config editing on puppet1 :)
16:09:31  * jima will test this zone before pushing it
16:09:40 -!- stickster_work is now known as stickster_afk
16:09:48 < mmcgrath> ok, next ticket #170
16:09:58 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/170
16:10:00 < mmcgrathbot> mmcgrath: #170 (Hosting respins) - Fedora Infrastructure - Trac
16:10:01 -!- tibbs [i=tibbs@fedora/tibbs] has quit "Konversation terminated!"
16:10:18 < mmcgrath> I've been trying to find sponsorship for a new torrent server.
16:10:26 < mmcgrath> In the meantime I'd like us to create spins.fedoraproject.org
16:10:31 < skvidal> with an assload of disk space?
16:10:50 < mmcgrath> skvidal: I'm requesting 500G-1T of space + average throughput of 100Mb
16:10:57 < mmcgrath> err maximum throughput of 100Mb.
16:10:57 < skvidal> good
16:11:14 < mmcgrath> The problem I've been running into lately is getting the presentation layer of this 
                     stuff going.
16:11:25 < mmcgrath> I've hit up the art people and the websites list and we just don't have many actual 
                     designers out there.
16:11:53 < mmcgrath> mizmo has always come through for us in the past but I'd like to find another 
                     avenue, mostly because she's busy and because I know there's just got to be other 
                     people interested in getting involved in this stuff.
16:12:19 < mmcgrath> Anyone have any ideas on this?
16:12:38 < mmcgrath> I got 2 responses from the art team and followed up a few times with them but have 
                     yet to hear back about any proof of concepts or anything like that.
16:12:44 < ivazquez> I can help. Let me hit the -websites archive to see what's there.
16:13:21 < mmcgrath> ivazquez: thanks, we're actually moving to a templated based system with genshi 
                     soon so that should help.  I'm just surprised at how little interst there is in 
                     this stuff.  I'd think web designers would be all over it.
16:13:39  * ivazquez needs to learn Genshi anyways
16:14:02 < mmcgrath> thats really all I've got on #170.  its an F8 milestone and available to anyone 
                     who'd be interested in doing the work to set it up.
16:14:03  * jima sucks at design, like many sysadmins.
16:14:09  * mmcgrath does too.
16:14:19 < mmcgrath> jima: knowing you suck is half the battle :)
16:14:23 < mmcgrath> well, for us its the whole battle
16:14:37 < mmcgrath> which reminds me
16:14:38 < mmcgrath> f13: ping?
16:14:42 < mmcgrath> anyone from rel-eng ping?
16:14:49  * notting can play rel-eng on irc
16:15:13 < mmcgrath> notting: how far along has the gaming spin gotten?
16:15:19 < mmcgrath> as far as being blessed by releng?
16:15:22 < notting> iirc, jeremy was doing test spins
16:15:29 < mmcgrath> jeremy: ping?
16:15:44 < jeremy> I gave it a try yesterday ... had to fix up a few things, but nothing too bad
16:15:46  * mmcgrath just wants to make sure to use the gaming spin as a sort of template for how future 
          spins should go.
16:15:56 < notting> jeremy: so, it's doing the desktop stuff right now?
16:15:57 < mmcgrath> jeremy: do you suspect it will be ready prior to F8?
16:16:01 < jeremy> notting: yes
16:16:05 -!- cyberpear [n=cyberpea@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:16:09 < jeremy> mmcgrath: yep.  and potentially an xfce one too
16:16:16 < mmcgrath> excellent news.
16:16:42 < mmcgrath> so when those are ready ready I'll probably have to sit down with someone and get 
                     the workflow figured out.  I don't think there's anything too fancy going on there.
16:17:13 < mmcgrath> Ok, anyone have any questions regarding the respins?
16:17:20  * warren now has debuginfo for dircproxy!
16:17:22 < warren> restarting...
16:17:50 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has quit "Terminated with extreme prejudice 
          - dircproxy 1.2.0"
16:17:53 < mmcgrath> ok, on to the schedule
16:17:58 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Schedule
16:18:04 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/Schedule
16:18:29 < mmcgrath> Corporate Sponsorship - Nothing new to report here.
16:18:49 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has joined #fedora-meeting
16:19:02 < mmcgrath> I've been in discussions with Peer1.com, we've had quite a few meetings now.  The 
                     last one was with their engineers so I'm hopeful to hear something soon.
16:19:09 < mmcgrath> either a yes or a no at least.
16:19:29 < mmcgrath> thats really it on that front.  We've got funding to ship a server to the new 
                     german colo but I doubt that will be ready for F8.
16:19:39 < mmcgrath> Any questions regarding Corporate sponsorship?  Any new leads?
16:19:51 < skvidal> mmcgrath: jeff_s
16:19:52 < mdomsch> mmcgrath, you have netapp on your list?
16:20:10 < mmcgrath> skvidal: will send an email soon
16:20:28 < mmcgrath> mdomsch: not yet, I think thats my fault, you gave me a contact over there I think 
                     but I never followed up.
16:20:49  * mmcgrath remembers now, I asked Chris Smith how much maintanence on our current netapp was 
          but never heard back.
16:21:17 < mdomsch> I spoke to Trond at LKS a few weeks ago, he seemed to think it was possible
16:21:33 < mmcgrath> mdomsch: I know you talked with one of the netapp guys, could you send me another 
                     email with all of his contact info?
16:21:39 < mdomsch> will do
16:21:46 < mmcgrath> much appreciated.
16:22:04 < mmcgrath> Anyone have anything else?
16:22:11 -!- paulobanon_ [n=psantos@xxxxxxxxxxxxxxx] has joined #fedora-meeting
16:22:23 < mmcgrath> Ok, moving on
16:22:23  * paulobanon_ is here now :)
16:22:25 < mmcgrath> paulobanon_: hey
16:22:32 < paulobanon_> back from our colo in DE :)
16:22:33 < mmcgrath> Nothing new on the architecutral documentation
16:22:45  * jima cheers paulobanon_ on
16:22:51 < mmcgrath> paulobanon_: the one thats also soon to be Fedora's colo in DE?
16:22:56 < paulobanon_> yup
16:23:08 < mmcgrath> solid
16:23:13 < mmcgrath> Nothing new in the SOP front
16:23:22 < mmcgrath> jima: did you apply for the syadmin-devel group?
16:23:33 < jima> mmcgrath: yeah, abadger1999 approved me
16:23:46 < mmcgrath> cool
16:23:55 < mmcgrath> so jima's got new sponsorship in the sysadmin-devel group.
16:24:13 < mmcgrath> Thats it for the schedule, I've got two other things I'd like to talk about before 
                     we open the floor.
16:24:21  * jima hides
16:24:26 < mmcgrath> one is another ticket.
16:24:44 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Misc Items
16:24:46  * warren OK, gdb is now watching this process.
16:24:49 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/93
16:24:51 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:25:11 < mmcgrath> warren: cool
16:25:15 < mmcgrath> So Fedora 8 is on the way
16:25:43 < mmcgrath> The current schedule says its going to be on November 8th.
16:25:55 < skvidal> mmcgrath: where will you be on nov 8th?
16:26:10 < mmcgrath> For right now that means that our Infrastructure freeze is on November 1st.
16:26:20 < mmcgrath> skvidal: I believe I'll still be here on November 8th but not on the 9th.
16:26:24 < skvidal> :)
16:26:37 < mmcgrath> So here's a roundup of where we are right this second.
16:26:55 < mmcgrath> We've got two proxy servers being load balanced in PHX each with 1G ram and each 
                     with 1 bad drive in it.
16:27:02 < mmcgrath> I've got a ticket in to replace each of those bad drives.
16:27:35 < mmcgrath> At present we have proxy3 in Denver with tummy.com.  Its got 5G ram dedicated to it.
16:27:49 < mmcgrath> And good drives, you get the idea.
16:27:50 < skvidal> odds on the disks being swapped out before f8?
16:27:59 < mmcgrath> skvidal: fair to good
16:28:08 < skvidal> mmcgrath: got an eta at all, yet?
16:28:10 < mmcgrath> its going to be one of stacy's final fairwells.
16:28:11 < mmcgrath> not yet.
16:28:14 < skvidal> ok
16:28:16 < skvidal> cool
16:28:20 < paulobanon_> oods on proxy4 before f8 ?
16:28:28 < paulobanon_> *odds
16:28:34 < mmcgrath> paulobanon_: not good, I just don't think we'll have one in before the 
                     infrastructure freeze.
16:28:43 < mmcgrath> having said that though, I don't think we'll *need* one.
16:28:46 < paulobanon_> oh well
16:29:08 < mmcgrath> I've also requested at the same time as the drive swaps happen that we make sure 
                     that some of the xen servers have a nic in the proxy network and the non-proxy 
                     network.
16:29:17 < mmcgrath> so we should be able to create proxy servers for the release in phx.
16:29:41 < mdomsch> and websites will keep the size of the front page small again, yes?
16:29:44 < jima> mmcgrath: i've done dual-nic setups with xen, yeah
16:29:46 < mmcgrath> I'm also thinking about adding an additional IP address to the tummy.com server so 
                     that twice as much traffic ends up there on release day.  My experience is that its 
                     just more peppy and with the RAM it will be better.
16:29:58 < mmcgrath> mdomsch: yes, probably very similar to what it is right now.
16:30:04 < mdomsch> goodness
16:30:19 < mmcgrath> so really, we could be in better shape, but we have a lot of options.
16:30:29 < mmcgrath> Something to note right now is that only fedoraproject.org email is going through 
                     proxy3.
16:30:43 < mmcgrath> not docs, not admin, not translate, not hosted, not koji, etc.
16:30:43 < skvidal> no other mx?
16:30:47 < skvidal> oh, I see
16:30:50 < mmcgrath> errrr sorry
16:30:53 < mmcgrath> s/email/web traffic/
16:30:56  * nirik is glad proxy3 is working out nicely. If you need anything on it, let me or jafo know. 
16:31:14 < jima> mmcgrath: you have no idea how confused you just made me :)
16:31:15 < mmcgrath> nirik: honestly it could very well save our butts this release so its going very 
                     well.  Thank you.
16:31:19 < mmcgrath> jima: sorry about that.
16:31:42 < mmcgrath> One thing I'd like to do is migrate docs over to fp.o.  That should be as simple as 
                     changing the CNAME and making sure that docs.fp.o is listening to "fp.o"
16:31:49 < mmcgrath> Those are our two major hitters.
16:32:00 < mmcgrath> Also this time around we have cachign enabled on the proxy servers.
16:32:19 < mmcgrath> We can alter it so that getfedora and the release notes are all served from memory.
16:32:27 < paulobanon_> yup
16:32:37 < mmcgrath> This stuff is all a little ways off but its good to start talking about it now.
16:32:48 < abadger1999> Thanks paulo!
16:32:53 < paulobanon_> and if any TG app wants it also, dont forget the /static/ :)
16:33:00 < mmcgrath> Does anyone else have any questions related to the F8 release?  I've added it to 
                     the meeting notes so we'll be discussing it from here on out.
16:33:10 < mmcgrath> paulobanon_: did we figure out the whole session issues with that?
16:33:38 < mmcgrath> paulobanon_: sorry we can talk about that later, probably not that important for 
                     this meeting.
16:33:46 < mmcgrath> The other thing I wanted to talk about is our mirrors setup
16:33:47 < mmcgrath> f13: ping
16:33:49 < mmcgrath> lmacken: ping?
16:34:00 < lmacken> mmcgrath: pong
16:34:08 < paulobanon_> mmcgrath: we can talk about that after the meeting in #fedora-admin
16:34:43 < mmcgrath> lmacken: so we've had a lot of problems with the mirrors over the last month, we've 
                     blown out updates 2 times.  the test3 release got copied to the wrong spot, test2 
                     is no longer available and the comps file that shipped last night was bad somehow.
16:34:54 < mmcgrath> I know some of that was our tools and some of that was a typo
16:34:59 < mmcgrath> and lastnights thing we just don't know about.
16:35:10 < jima> yeah :(
16:35:10 -!- Lovechild [n=david@about/unixlove/Lovechild] has quit "Ex-Chat"
16:35:37 < mmcgrath> is there anything that we can be doing right now to make sure this stuf doesn't 
                     happen anymore?  I know you're working on mash, will that fix all of these issues?
16:36:12 < mmcgrath> do we need to dedicate more resources to it?
16:36:13 < lmacken> well, we don't have much control with what IS pushes out.. but I think it couldn't 
                    hurt to add a bit more paranoia to our tools before they flip the live bits.
16:36:24 < paulobanon_> were all problems in the tools been identified ?
16:36:30 < lmacken> even with all of the sanity checking bodhi could have done, it wouldn't have stopped 
                    what happened today with the comps..
16:36:42 < jima> how do we push bits out? rsync?
16:36:51 < lmacken> the repo that bodhi mashed was fine, but it synced out with a bad comps
16:36:57 < mmcgrath> lmacken: can't we add a check to bodhi does the sha1sum match whats in the xml file?
16:37:09 < lmacken> I haven't worked on mash at all..
16:37:14 < mmcgrath> <nod>
16:37:20 < lmacken> mmcgrath: yeah.. but that wouldn't have helped today, since everything matched up 
                    fine
16:00:06  * jima here
16:00:25  * mmcgrath looks for the two to drop
16:00:36 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:00:38 < mmcgrath> Who's here?
16:00:42  * lmacken 
16:00:42 -!- warren [i=warren@redhat/wombat/warren] has quit Remote closed the connection
16:00:42 -!- jeremy [i=katzj@nat/redhat/x-824cfb21e0d420e3] has quit Remote closed the connection
16:00:46 < jima> ha!
16:00:53 < ivazquez> Pong.
16:01:22 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has joined #fedora-meeting
16:01:23 < mmcgrath> dgilmore: paulobanon abadger1999 mbacovsk f13 skvidal jima + anyone I've forgotten
16:01:27 < mmcgrath> ping
16:01:36 < skvidal> pong
16:01:37 < frankc> Frank is eavesdropping
16:01:40 < abadger1999> pong
16:01:41 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has joined #fedora-meeting
16:01:41 < skvidal> and there go warren and jeremy
16:01:44 -!- |DrJef| [n=onefjef@fedora/Jef] has joined #fedora-meeting
16:01:47 < mmcgrath> :)
16:02:09 < warren> mmcgrath, damn you!
16:02:15 < mmcgrath> I didn't do it I swear!!!
16:02:17  * jima refrains from commentary on frank :)
16:02:22 < mmcgrath> heh
16:02:29 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
16:02:33 < mmcgrath> .tiny 
https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority
16:02:33 < mmcgrathbot> mmcgrath: http://tinyurl.com/yth34b
16:02:37 < jima> mmcgrath: could it be the topic change?
16:02:39 < warren> mmcgrath, do you change the topic or something?
16:02:41 < warren> I should put it under gdb...
16:02:52 -!- skvidal changed the topic of #fedora-meeting to: Infrastructure -- Tickets test
16:02:53 < mmcgrath> warren: it could be, after the meeting I'll change it again and see what happens.
16:03:00 < skvidal> nope, they didn't die
16:03:01 < warren> mmcgrath, k
16:03:03 < jima> warren: it was immediately after the topic change
16:03:08 < lmacken> lets start the meeting over and see if it happens again ;P
16:03:18 -!- jima changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:03:19 < warren> waiit....I don't have debuginfo
16:03:26 < mmcgrath> ok, first ticket relates to DNS -
16:03:30 < lmacken> (but not really)
16:03:33 < mmcgrath> #154
16:03:48 < mmcgrath> Jima spent some time this afternoon to create publictest[1-9].fedoraproject.org
16:03:51 < jima> bizarre.
16:03:58 < jima> mmcgrath: yeah, like 30 seconds. ;)
16:04:18 < lmacken> cool
16:04:27 < mmcgrath> This is mostly just so we can A) be consistent with the rest of our services and so 
                     we don't have to run DNS requests through RHIS.
16:04:42 < mmcgrath> I'll be setting up a RHEL box soon (I was waiting for this part to be done)
16:04:42 < jima> would anyone be too mad if i cleaned up the fp.o zone file? like, alphabetized it? or 
                 would that be too cvs-noisy?
16:04:58 < lmacken> is bastion the only one left to have .fedora.redhat.com ?
16:05:01 < mmcgrath> jima: actually I'd appreciate that, keep the 'balanced' pieces in their own section.
16:05:26 < mmcgrath> lmacken: I think so, there's a few things out there, like cvs.fedora.redhat.com 
                     still exists but so does cvs.fedoraproject.org
16:05:33 < jima> mmcgrath: that's fine, but "The rest of our configs"?
16:05:34 < mmcgrath> jima: while you're in there add a bastion.fedoraproject.org
16:05:58 < skvidal> jima: umm - but one thing
16:06:03 < skvidal> jima: test it, A LOT
16:06:05 < skvidal> before you push it
16:06:07 < jima> sections are fine, all the random hostnames are just annoying :)
16:06:15 < skvidal> b/c clipping dns is a really good way to cause a lot of noise
16:06:17 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
16:06:18 < mmcgrath> jima: I agree.
16:06:21 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:06:39 < jima> skvidal: enh...dns administration is a big part of my job.
16:06:43 < skvidal> cool
16:06:45 < skvidal> just making sure
16:06:57 < jima> i haven't made a dns boo-boo in YEARS :)
16:07:04  * nirik notes that named-checkzone is a handy command. ;) 
16:07:13 < jima> (that was 1998, when i put in a #comment in a zone file. yes, seriously.)
16:07:15 < skvidal> jima: no time like the present to start :)
16:07:23 < mmcgrath> <nod>
16:07:42 < mmcgrath> jima: so yeah I guess the config will have two sections both alphabetized.
16:07:44 < jima> *sigh* if only named-checkzone were on puppet1 :)
16:08:31 < mmcgrath> jima: you've got a workstation :-P
16:08:35 < nirik> it's in the main bind package along with named-checkconf (which is also very handy)
16:09:01 < mmcgrath> Anyone have any other questions regarding the DNS stuff thats going on?  I doubt 
                     that we'll just cut over to using it but we'll probably grandfather it in.
16:09:07 < jima> mmcgrath: yeah, i know -- i just do all my config editing on puppet1 :)
16:09:31  * jima will test this zone before pushing it
16:09:40 -!- stickster_work is now known as stickster_afk
16:09:48 < mmcgrath> ok, next ticket #170
16:09:58 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/170
16:10:00 < mmcgrathbot> mmcgrath: #170 (Hosting respins) - Fedora Infrastructure - Trac
16:10:01 -!- tibbs [i=tibbs@fedora/tibbs] has quit "Konversation terminated!"
16:10:18 < mmcgrath> I've been trying to find sponsorship for a new torrent server.
16:10:26 < mmcgrath> In the meantime I'd like us to create spins.fedoraproject.org
16:10:31 < skvidal> with an assload of disk space?
16:10:50 < mmcgrath> skvidal: I'm requesting 500G-1T of space + average throughput of 100Mb
16:10:57 < mmcgrath> err maximum throughput of 100Mb.
16:10:57 < skvidal> good
16:11:14 < mmcgrath> The problem I've been running into lately is getting the presentation layer of this 
                     stuff going.
16:11:25 < mmcgrath> I've hit up the art people and the websites list and we just don't have many actual 
                     designers out there.
16:11:53 < mmcgrath> mizmo has always come through for us in the past but I'd like to find another 
                     avenue, mostly because she's busy and because I know there's just got to be other 
                     people interested in getting involved in this stuff.
16:12:19 < mmcgrath> Anyone have any ideas on this?
16:12:38 < mmcgrath> I got 2 responses from the art team and followed up a few times with them but have 
                     yet to hear back about any proof of concepts or anything like that.
16:12:44 < ivazquez> I can help. Let me hit the -websites archive to see what's there.
16:13:21 < mmcgrath> ivazquez: thanks, we're actually moving to a templated based system with genshi 
                     soon so that should help.  I'm just surprised at how little interst there is in 
                     this stuff.  I'd think web designers would be all over it.
16:13:39  * ivazquez needs to learn Genshi anyways
16:14:02 < mmcgrath> thats really all I've got on #170.  its an F8 milestone and available to anyone 
                     who'd be interested in doing the work to set it up.
16:14:03  * jima sucks at design, like many sysadmins.
16:14:09  * mmcgrath does too.
16:14:19 < mmcgrath> jima: knowing you suck is half the battle :)
16:14:23 < mmcgrath> well, for us its the whole battle
16:14:37 < mmcgrath> which reminds me
16:14:38 < mmcgrath> f13: ping?
16:14:42 < mmcgrath> anyone from rel-eng ping?
16:14:49  * notting can play rel-eng on irc
16:15:13 < mmcgrath> notting: how far along has the gaming spin gotten?
16:15:19 < mmcgrath> as far as being blessed by releng?
16:15:22 < notting> iirc, jeremy was doing test spins
16:15:29 < mmcgrath> jeremy: ping?
16:15:44 < jeremy> I gave it a try yesterday ... had to fix up a few things, but nothing too bad
16:15:46  * mmcgrath just wants to make sure to use the gaming spin as a sort of template for how future 
          spins should go.
16:15:56 < notting> jeremy: so, it's doing the desktop stuff right now?
16:15:57 < mmcgrath> jeremy: do you suspect it will be ready prior to F8?
16:16:01 < jeremy> notting: yes
16:16:05 -!- cyberpear [n=cyberpea@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:16:09 < jeremy> mmcgrath: yep.  and potentially an xfce one too
16:16:16 < mmcgrath> excellent news.
16:16:42 < mmcgrath> so when those are ready ready I'll probably have to sit down with someone and get 
                     the workflow figured out.  I don't think there's anything too fancy going on there.
16:17:13 < mmcgrath> Ok, anyone have any questions regarding the respins?
16:17:20  * warren now has debuginfo for dircproxy!
16:17:22 < warren> restarting...
16:17:50 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has quit "Terminated with extreme prejudice 
          - dircproxy 1.2.0"
16:17:53 < mmcgrath> ok, on to the schedule
16:17:58 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Schedule
16:18:04 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/Schedule
16:18:29 < mmcgrath> Corporate Sponsorship - Nothing new to report here.
16:18:49 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has joined #fedora-meeting
16:19:02 < mmcgrath> I've been in discussions with Peer1.com, we've had quite a few meetings now.  The 
                     last one was with their engineers so I'm hopeful to hear something soon.
16:19:09 < mmcgrath> either a yes or a no at least.
16:19:29 < mmcgrath> thats really it on that front.  We've got funding to ship a server to the new 
                     german colo but I doubt that will be ready for F8.
16:19:39 < mmcgrath> Any questions regarding Corporate sponsorship?  Any new leads?
16:19:51 < skvidal> mmcgrath: jeff_s
16:19:52 < mdomsch> mmcgrath, you have netapp on your list?
16:20:10 < mmcgrath> skvidal: will send an email soon
16:20:28 < mmcgrath> mdomsch: not yet, I think thats my fault, you gave me a contact over there I think 
                     but I never followed up.
16:20:49  * mmcgrath remembers now, I asked Chris Smith how much maintanence on our current netapp was 
          but never heard back.
16:21:17 < mdomsch> I spoke to Trond at LKS a few weeks ago, he seemed to think it was possible
16:21:33 < mmcgrath> mdomsch: I know you talked with one of the netapp guys, could you send me another 
                     email with all of his contact info?
16:21:39 < mdomsch> will do
16:21:46 < mmcgrath> much appreciated.
16:22:04 < mmcgrath> Anyone have anything else?
16:22:11 -!- paulobanon_ [n=psantos@xxxxxxxxxxxxxxx] has joined #fedora-meeting
16:22:23 < mmcgrath> Ok, moving on
16:22:23  * paulobanon_ is here now :)
16:22:25 < mmcgrath> paulobanon_: hey
16:22:32 < paulobanon_> back from our colo in DE :)
16:22:33 < mmcgrath> Nothing new on the architecutral documentation
16:22:45  * jima cheers paulobanon_ on
16:22:51 < mmcgrath> paulobanon_: the one thats also soon to be Fedora's colo in DE?
16:22:56 < paulobanon_> yup
16:23:08 < mmcgrath> solid
16:23:13 < mmcgrath> Nothing new in the SOP front
16:23:22 < mmcgrath> jima: did you apply for the syadmin-devel group?
16:23:33 < jima> mmcgrath: yeah, abadger1999 approved me
16:23:46 < mmcgrath> cool
16:23:55 < mmcgrath> so jima's got new sponsorship in the sysadmin-devel group.
16:24:13 < mmcgrath> Thats it for the schedule, I've got two other things I'd like to talk about before 
                     we open the floor.
16:24:21  * jima hides
16:24:26 < mmcgrath> one is another ticket.
16:24:44 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Misc Items
16:24:46  * warren OK, gdb is now watching this process.
16:24:49 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/93
16:24:51 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:25:11 < mmcgrath> warren: cool
16:25:15 < mmcgrath> So Fedora 8 is on the way
16:25:43 < mmcgrath> The current schedule says its going to be on November 8th.
16:25:55 < skvidal> mmcgrath: where will you be on nov 8th?
16:26:10 < mmcgrath> For right now that means that our Infrastructure freeze is on November 1st.
16:26:20 < mmcgrath> skvidal: I believe I'll still be here on November 8th but not on the 9th.
16:26:24 < skvidal> :)
16:26:37 < mmcgrath> So here's a roundup of where we are right this second.
16:26:55 < mmcgrath> We've got two proxy servers being load balanced in PHX each with 1G ram and each 
                     with 1 bad drive in it.
16:27:02 < mmcgrath> I've got a ticket in to replace each of those bad drives.
16:27:35 < mmcgrath> At present we have proxy3 in Denver with tummy.com.  Its got 5G ram dedicated to it.
16:27:49 < mmcgrath> And good drives, you get the idea.
16:27:50 < skvidal> odds on the disks being swapped out before f8?
16:27:59 < mmcgrath> skvidal: fair to good
16:28:08 < skvidal> mmcgrath: got an eta at all, yet?
16:28:10 < mmcgrath> its going to be one of stacy's final fairwells.
16:28:11 < mmcgrath> not yet.
16:28:14 < skvidal> ok
16:28:16 < skvidal> cool
16:28:20 < paulobanon_> oods on proxy4 before f8 ?
16:28:28 < paulobanon_> *odds
16:28:34 < mmcgrath> paulobanon_: not good, I just don't think we'll have one in before the 
                     infrastructure freeze.
16:28:43 < mmcgrath> having said that though, I don't think we'll *need* one.
16:28:46 < paulobanon_> oh well
16:29:08 < mmcgrath> I've also requested at the same time as the drive swaps happen that we make sure 
                     that some of the xen servers have a nic in the proxy network and the non-proxy 
                     network.
16:29:17 < mmcgrath> so we should be able to create proxy servers for the release in phx.
16:29:41 < mdomsch> and websites will keep the size of the front page small again, yes?
16:29:44 < jima> mmcgrath: i've done dual-nic setups with xen, yeah
16:29:46 < mmcgrath> I'm also thinking about adding an additional IP address to the tummy.com server so 
                     that twice as much traffic ends up there on release day.  My experience is that its 
                     just more peppy and with the RAM it will be better.
16:29:58 < mmcgrath> mdomsch: yes, probably very similar to what it is right now.
16:30:04 < mdomsch> goodness
16:30:19 < mmcgrath> so really, we could be in better shape, but we have a lot of options.
16:30:29 < mmcgrath> Something to note right now is that only fedoraproject.org email is going through 
                     proxy3.
16:30:43 < mmcgrath> not docs, not admin, not translate, not hosted, not koji, etc.
16:30:43 < skvidal> no other mx?
16:30:47 < skvidal> oh, I see
16:30:50 < mmcgrath> errrr sorry
16:30:53 < mmcgrath> s/email/web traffic/
16:30:56  * nirik is glad proxy3 is working out nicely. If you need anything on it, let me or jafo know. 
16:31:14 < jima> mmcgrath: you have no idea how confused you just made me :)
16:31:15 < mmcgrath> nirik: honestly it could very well save our butts this release so its going very 
                     well.  Thank you.
16:31:19 < mmcgrath> jima: sorry about that.
16:31:42 < mmcgrath> One thing I'd like to do is migrate docs over to fp.o.  That should be as simple as 
                     changing the CNAME and making sure that docs.fp.o is listening to "fp.o"
16:31:49 < mmcgrath> Those are our two major hitters.
16:32:00 < mmcgrath> Also this time around we have cachign enabled on the proxy servers.
16:32:19 < mmcgrath> We can alter it so that getfedora and the release notes are all served from memory.
16:32:27 < paulobanon_> yup
16:32:37 < mmcgrath> This stuff is all a little ways off but its good to start talking about it now.
16:32:48 < abadger1999> Thanks paulo!
16:32:53 < paulobanon_> and if any TG app wants it also, dont forget the /static/ :)
16:33:00 < mmcgrath> Does anyone else have any questions related to the F8 release?  I've added it to 
                     the meeting notes so we'll be discussing it from here on out.
16:33:10 < mmcgrath> paulobanon_: did we figure out the whole session issues with that?
16:33:38 < mmcgrath> paulobanon_: sorry we can talk about that later, probably not that important for 
                     this meeting.
16:33:46 < mmcgrath> The other thing I wanted to talk about is our mirrors setup
16:33:47 < mmcgrath> f13: ping
16:33:49 < mmcgrath> lmacken: ping?
16:34:00 < lmacken> mmcgrath: pong
16:34:08 < paulobanon_> mmcgrath: we can talk about that after the meeting in #fedora-admin
16:34:43 < mmcgrath> lmacken: so we've had a lot of problems with the mirrors over the last month, we've 
                     blown out updates 2 times.  the test3 release got copied to the wrong spot, test2 
                     is no longer available and the comps file that shipped last night was bad somehow.
16:34:54 < mmcgrath> I know some of that was our tools and some of that was a typo
16:34:59 < mmcgrath> and lastnights thing we just don't know about.
16:35:10 < jima> yeah :(
16:35:10 -!- Lovechild [n=david@about/unixlove/Lovechild] has quit "Ex-Chat"
16:35:37 < mmcgrath> is there anything that we can be doing right now to make sure this stuf doesn't 
                     happen anymore?  I know you're working on mash, will that fix all of these issues?
16:36:12 < mmcgrath> do we need to dedicate more resources to it?
16:36:13 < lmacken> well, we don't have much control with what IS pushes out.. but I think it couldn't 
                    hurt to add a bit more paranoia to our tools before they flip the live bits.
16:36:24 < paulobanon_> were all problems in the tools been identified ?
16:36:30 < lmacken> even with all of the sanity checking bodhi could have done, it wouldn't have stopped 
                    what happened today with the comps..
16:36:42 < jima> how do we push bits out? rsync?
16:36:51 < lmacken> the repo that bodhi mashed was fine, but it synced out with a bad comps
16:36:57 < mmcgrath> lmacken: can't we add a check to bodhi does the sha1sum match whats in the xml file?
16:37:09 < lmacken> I haven't worked on mash at all..
16:37:14 < mmcgrath> <nod>
16:37:20 < lmacken> mmcgrath: yeah.. but that wouldn't have helped today, since everything matched up 
                    fine
16:00:06  * jima here
16:00:25  * mmcgrath looks for the two to drop
16:00:36 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:00:38 < mmcgrath> Who's here?
16:00:42  * lmacken 
16:00:42 -!- warren [i=warren@redhat/wombat/warren] has quit Remote closed the connection
16:00:42 -!- jeremy [i=katzj@nat/redhat/x-824cfb21e0d420e3] has quit Remote closed the connection
16:00:46 < jima> ha!
16:00:53 < ivazquez> Pong.
16:01:22 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has joined #fedora-meeting
16:01:23 < mmcgrath> dgilmore: paulobanon abadger1999 mbacovsk f13 skvidal jima + anyone I've forgotten
16:01:27 < mmcgrath> ping
16:01:36 < skvidal> pong
16:01:37 < frankc> Frank is eavesdropping
16:01:40 < abadger1999> pong
16:01:41 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has joined #fedora-meeting
16:01:41 < skvidal> and there go warren and jeremy
16:01:44 -!- |DrJef| [n=onefjef@fedora/Jef] has joined #fedora-meeting
16:01:47 < mmcgrath> :)
16:02:09 < warren> mmcgrath, damn you!
16:02:15 < mmcgrath> I didn't do it I swear!!!
16:02:17  * jima refrains from commentary on frank :)
16:02:22 < mmcgrath> heh
16:02:29 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Tickets
16:02:33 < mmcgrath> .tiny 
https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=%7EMeeting&order=priority
16:02:33 < mmcgrathbot> mmcgrath: http://tinyurl.com/yth34b
16:02:37 < jima> mmcgrath: could it be the topic change?
16:02:39 < warren> mmcgrath, do you change the topic or something?
16:02:41 < warren> I should put it under gdb...
16:02:52 -!- skvidal changed the topic of #fedora-meeting to: Infrastructure -- Tickets test
16:02:53 < mmcgrath> warren: it could be, after the meeting I'll change it again and see what happens.
16:03:00 < skvidal> nope, they didn't die
16:03:01 < warren> mmcgrath, k
16:03:03 < jima> warren: it was immediately after the topic change
16:03:08 < lmacken> lets start the meeting over and see if it happens again ;P
16:03:18 -!- jima changed the topic of #fedora-meeting to: Infrastructure -- Role Call
16:03:19 < warren> waiit....I don't have debuginfo
16:03:26 < mmcgrath> ok, first ticket relates to DNS -
16:03:30 < lmacken> (but not really)
16:03:33 < mmcgrath> #154
16:03:48 < mmcgrath> Jima spent some time this afternoon to create publictest[1-9].fedoraproject.org
16:03:51 < jima> bizarre.
16:03:58 < jima> mmcgrath: yeah, like 30 seconds. ;)
16:04:18 < lmacken> cool
16:04:27 < mmcgrath> This is mostly just so we can A) be consistent with the rest of our services and so 
                     we don't have to run DNS requests through RHIS.
16:04:42 < mmcgrath> I'll be setting up a RHEL box soon (I was waiting for this part to be done)
16:04:42 < jima> would anyone be too mad if i cleaned up the fp.o zone file? like, alphabetized it? or 
                 would that be too cvs-noisy?
16:04:58 < lmacken> is bastion the only one left to have .fedora.redhat.com ?
16:05:01 < mmcgrath> jima: actually I'd appreciate that, keep the 'balanced' pieces in their own section.
16:05:26 < mmcgrath> lmacken: I think so, there's a few things out there, like cvs.fedora.redhat.com 
                     still exists but so does cvs.fedoraproject.org
16:05:33 < jima> mmcgrath: that's fine, but "The rest of our configs"?
16:05:34 < mmcgrath> jima: while you're in there add a bastion.fedoraproject.org
16:05:58 < skvidal> jima: umm - but one thing
16:06:03 < skvidal> jima: test it, A LOT
16:06:05 < skvidal> before you push it
16:06:07 < jima> sections are fine, all the random hostnames are just annoying :)
16:06:15 < skvidal> b/c clipping dns is a really good way to cause a lot of noise
16:06:17 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
16:06:18 < mmcgrath> jima: I agree.
16:06:21 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:06:39 < jima> skvidal: enh...dns administration is a big part of my job.
16:06:43 < skvidal> cool
16:06:45 < skvidal> just making sure
16:06:57 < jima> i haven't made a dns boo-boo in YEARS :)
16:07:04  * nirik notes that named-checkzone is a handy command. ;) 
16:07:13 < jima> (that was 1998, when i put in a #comment in a zone file. yes, seriously.)
16:07:15 < skvidal> jima: no time like the present to start :)
16:07:23 < mmcgrath> <nod>
16:07:42 < mmcgrath> jima: so yeah I guess the config will have two sections both alphabetized.
16:07:44 < jima> *sigh* if only named-checkzone were on puppet1 :)
16:08:31 < mmcgrath> jima: you've got a workstation :-P
16:08:35 < nirik> it's in the main bind package along with named-checkconf (which is also very handy)
16:09:01 < mmcgrath> Anyone have any other questions regarding the DNS stuff thats going on?  I doubt 
                     that we'll just cut over to using it but we'll probably grandfather it in.
16:09:07 < jima> mmcgrath: yeah, i know -- i just do all my config editing on puppet1 :)
16:09:31  * jima will test this zone before pushing it
16:09:40 -!- stickster_work is now known as stickster_afk
16:09:48 < mmcgrath> ok, next ticket #170
16:09:58 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/170
16:10:00 < mmcgrathbot> mmcgrath: #170 (Hosting respins) - Fedora Infrastructure - Trac
16:10:01 -!- tibbs [i=tibbs@fedora/tibbs] has quit "Konversation terminated!"
16:10:18 < mmcgrath> I've been trying to find sponsorship for a new torrent server.
16:10:26 < mmcgrath> In the meantime I'd like us to create spins.fedoraproject.org
16:10:31 < skvidal> with an assload of disk space?
16:10:50 < mmcgrath> skvidal: I'm requesting 500G-1T of space + average throughput of 100Mb
16:10:57 < mmcgrath> err maximum throughput of 100Mb.
16:10:57 < skvidal> good
16:11:14 < mmcgrath> The problem I've been running into lately is getting the presentation layer of this 
                     stuff going.
16:11:25 < mmcgrath> I've hit up the art people and the websites list and we just don't have many actual 
                     designers out there.
16:11:53 < mmcgrath> mizmo has always come through for us in the past but I'd like to find another 
                     avenue, mostly because she's busy and because I know there's just got to be other 
                     people interested in getting involved in this stuff.
16:12:19 < mmcgrath> Anyone have any ideas on this?
16:12:38 < mmcgrath> I got 2 responses from the art team and followed up a few times with them but have 
                     yet to hear back about any proof of concepts or anything like that.
16:12:44 < ivazquez> I can help. Let me hit the -websites archive to see what's there.
16:13:21 < mmcgrath> ivazquez: thanks, we're actually moving to a templated based system with genshi 
                     soon so that should help.  I'm just surprised at how little interst there is in 
                     this stuff.  I'd think web designers would be all over it.
16:13:39  * ivazquez needs to learn Genshi anyways
16:14:02 < mmcgrath> thats really all I've got on #170.  its an F8 milestone and available to anyone 
                     who'd be interested in doing the work to set it up.
16:14:03  * jima sucks at design, like many sysadmins.
16:14:09  * mmcgrath does too.
16:14:19 < mmcgrath> jima: knowing you suck is half the battle :)
16:14:23 < mmcgrath> well, for us its the whole battle
16:14:37 < mmcgrath> which reminds me
16:14:38 < mmcgrath> f13: ping?
16:14:42 < mmcgrath> anyone from rel-eng ping?
16:14:49  * notting can play rel-eng on irc
16:15:13 < mmcgrath> notting: how far along has the gaming spin gotten?
16:15:19 < mmcgrath> as far as being blessed by releng?
16:15:22 < notting> iirc, jeremy was doing test spins
16:15:29 < mmcgrath> jeremy: ping?
16:15:44 < jeremy> I gave it a try yesterday ... had to fix up a few things, but nothing too bad
16:15:46  * mmcgrath just wants to make sure to use the gaming spin as a sort of template for how future 
          spins should go.
16:15:56 < notting> jeremy: so, it's doing the desktop stuff right now?
16:15:57 < mmcgrath> jeremy: do you suspect it will be ready prior to F8?
16:16:01 < jeremy> notting: yes
16:16:05 -!- cyberpear [n=cyberpea@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
16:16:09 < jeremy> mmcgrath: yep.  and potentially an xfce one too
16:16:16 < mmcgrath> excellent news.
16:16:42 < mmcgrath> so when those are ready ready I'll probably have to sit down with someone and get 
                     the workflow figured out.  I don't think there's anything too fancy going on there.
16:17:13 < mmcgrath> Ok, anyone have any questions regarding the respins?
16:17:20  * warren now has debuginfo for dircproxy!
16:17:22 < warren> restarting...
16:17:50 -!- warren [i=warren@nat/redhat/x-d40cd22677cdddea] has quit "Terminated with extreme prejudice 
          - dircproxy 1.2.0"
16:17:53 < mmcgrath> ok, on to the schedule
16:17:58 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Schedule
16:18:04 < mmcgrath> http://fedoraproject.org/wiki/Infrastructure/Schedule
16:18:29 < mmcgrath> Corporate Sponsorship - Nothing new to report here.
16:18:49 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has joined #fedora-meeting
16:19:02 < mmcgrath> I've been in discussions with Peer1.com, we've had quite a few meetings now.  The 
                     last one was with their engineers so I'm hopeful to hear something soon.
16:19:09 < mmcgrath> either a yes or a no at least.
16:19:29 < mmcgrath> thats really it on that front.  We've got funding to ship a server to the new 
                     german colo but I doubt that will be ready for F8.
16:19:39 < mmcgrath> Any questions regarding Corporate sponsorship?  Any new leads?
16:19:51 < skvidal> mmcgrath: jeff_s
16:19:52 < mdomsch> mmcgrath, you have netapp on your list?
16:20:10 < mmcgrath> skvidal: will send an email soon
16:20:28 < mmcgrath> mdomsch: not yet, I think thats my fault, you gave me a contact over there I think 
                     but I never followed up.
16:20:49  * mmcgrath remembers now, I asked Chris Smith how much maintanence on our current netapp was 
          but never heard back.
16:21:17 < mdomsch> I spoke to Trond at LKS a few weeks ago, he seemed to think it was possible
16:21:33 < mmcgrath> mdomsch: I know you talked with one of the netapp guys, could you send me another 
                     email with all of his contact info?
16:21:39 < mdomsch> will do
16:21:46 < mmcgrath> much appreciated.
16:22:04 < mmcgrath> Anyone have anything else?
16:22:11 -!- paulobanon_ [n=psantos@xxxxxxxxxxxxxxx] has joined #fedora-meeting
16:22:23 < mmcgrath> Ok, moving on
16:22:23  * paulobanon_ is here now :)
16:22:25 < mmcgrath> paulobanon_: hey
16:22:32 < paulobanon_> back from our colo in DE :)
16:22:33 < mmcgrath> Nothing new on the architecutral documentation
16:22:45  * jima cheers paulobanon_ on
16:22:51 < mmcgrath> paulobanon_: the one thats also soon to be Fedora's colo in DE?
16:22:56 < paulobanon_> yup
16:23:08 < mmcgrath> solid
16:23:13 < mmcgrath> Nothing new in the SOP front
16:23:22 < mmcgrath> jima: did you apply for the syadmin-devel group?
16:23:33 < jima> mmcgrath: yeah, abadger1999 approved me
16:23:46 < mmcgrath> cool
16:23:55 < mmcgrath> so jima's got new sponsorship in the sysadmin-devel group.
16:24:13 < mmcgrath> Thats it for the schedule, I've got two other things I'd like to talk about before 
                     we open the floor.
16:24:21  * jima hides
16:24:26 < mmcgrath> one is another ticket.
16:24:44 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Misc Items
16:24:46  * warren OK, gdb is now watching this process.
16:24:49 < mmcgrath> .title https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/93
16:24:51 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac
16:25:11 < mmcgrath> warren: cool
16:25:15 < mmcgrath> So Fedora 8 is on the way
16:25:43 < mmcgrath> The current schedule says its going to be on November 8th.
16:25:55 < skvidal> mmcgrath: where will you be on nov 8th?
16:26:10 < mmcgrath> For right now that means that our Infrastructure freeze is on November 1st.
16:26:20 < mmcgrath> skvidal: I believe I'll still be here on November 8th but not on the 9th.
16:26:24 < skvidal> :)
16:26:37 < mmcgrath> So here's a roundup of where we are right this second.
16:26:55 < mmcgrath> We've got two proxy servers being load balanced in PHX each with 1G ram and each 
                     with 1 bad drive in it.
16:27:02 < mmcgrath> I've got a ticket in to replace each of those bad drives.
16:27:35 < mmcgrath> At present we have proxy3 in Denver with tummy.com.  Its got 5G ram dedicated to it.
16:27:49 < mmcgrath> And good drives, you get the idea.
16:27:50 < skvidal> odds on the disks being swapped out before f8?
16:27:59 < mmcgrath> skvidal: fair to good
16:28:08 < skvidal> mmcgrath: got an eta at all, yet?
16:28:10 < mmcgrath> its going to be one of stacy's final fairwells.
16:28:11 < mmcgrath> not yet.
16:28:14 < skvidal> ok
16:28:16 < skvidal> cool
16:28:20 < paulobanon_> oods on proxy4 before f8 ?
16:28:28 < paulobanon_> *odds
16:28:34 < mmcgrath> paulobanon_: not good, I just don't think we'll have one in before the 
                     infrastructure freeze.
16:28:43 < mmcgrath> having said that though, I don't think we'll *need* one.
16:28:46 < paulobanon_> oh well
16:29:08 < mmcgrath> I've also requested at the same time as the drive swaps happen that we make sure 
                     that some of the xen servers have a nic in the proxy network and the non-proxy 
                     network.
16:29:17 < mmcgrath> so we should be able to create proxy servers for the release in phx.
16:29:41 < mdomsch> and websites will keep the size of the front page small again, yes?
16:29:44 < jima> mmcgrath: i've done dual-nic setups with xen, yeah
16:29:46 < mmcgrath> I'm also thinking about adding an additional IP address to the tummy.com server so 
                     that twice as much traffic ends up there on release day.  My experience is that its 
                     just more peppy and with the RAM it will be better.
16:29:58 < mmcgrath> mdomsch: yes, probably very similar to what it is right now.
16:30:04 < mdomsch> goodness
16:30:19 < mmcgrath> so really, we could be in better shape, but we have a lot of options.
16:30:29 < mmcgrath> Something to note right now is that only fedoraproject.org email is going through 
                     proxy3.
16:30:43 < mmcgrath> not docs, not admin, not translate, not hosted, not koji, etc.
16:30:43 < skvidal> no other mx?
16:30:47 < skvidal> oh, I see
16:30:50 < mmcgrath> errrr sorry
16:30:53 < mmcgrath> s/email/web traffic/
16:30:56  * nirik is glad proxy3 is working out nicely. If you need anything on it, let me or jafo know. 
16:31:14 < jima> mmcgrath: you have no idea how confused you just made me :)
16:31:15 < mmcgrath> nirik: honestly it could very well save our butts this release so its going very 
                     well.  Thank you.
16:31:19 < mmcgrath> jima: sorry about that.
16:31:42 < mmcgrath> One thing I'd like to do is migrate docs over to fp.o.  That should be as simple as 
                     changing the CNAME and making sure that docs.fp.o is listening to "fp.o"
16:31:49 < mmcgrath> Those are our two major hitters.
16:32:00 < mmcgrath> Also this time around we have cachign enabled on the proxy servers.
16:32:19 < mmcgrath> We can alter it so that getfedora and the release notes are all served from memory.
16:32:27 < paulobanon_> yup
16:32:37 < mmcgrath> This stuff is all a little ways off but its good to start talking about it now.
16:32:48 < abadger1999> Thanks paulo!
16:32:53 < paulobanon_> and if any TG app wants it also, dont forget the /static/ :)
16:33:00 < mmcgrath> Does anyone else have any questions related to the F8 release?  I've added it to 
                     the meeting notes so we'll be discussing it from here on out.
16:33:10 < mmcgrath> paulobanon_: did we figure out the whole session issues with that?
16:33:38 < mmcgrath> paulobanon_: sorry we can talk about that later, probably not that important for 
                     this meeting.
16:33:46 < mmcgrath> The other thing I wanted to talk about is our mirrors setup
16:33:47 < mmcgrath> f13: ping
16:33:49 < mmcgrath> lmacken: ping?
16:34:00 < lmacken> mmcgrath: pong
16:34:08 < paulobanon_> mmcgrath: we can talk about that after the meeting in #fedora-admin
16:34:43 < mmcgrath> lmacken: so we've had a lot of problems with the mirrors over the last month, we've 
                     blown out updates 2 times.  the test3 release got copied to the wrong spot, test2 
                     is no longer available and the comps file that shipped last night was bad somehow.
16:34:54 < mmcgrath> I know some of that was our tools and some of that was a typo
16:34:59 < mmcgrath> and lastnights thing we just don't know about.
16:35:10 < jima> yeah :(
16:35:10 -!- Lovechild [n=david@about/unixlove/Lovechild] has quit "Ex-Chat"
16:35:37 < mmcgrath> is there anything that we can be doing right now to make sure this stuf doesn't 
                     happen anymore?  I know you're working on mash, will that fix all of these issues?
16:36:12 < mmcgrath> do we need to dedicate more resources to it?
16:36:13 < lmacken> well, we don't have much control with what IS pushes out.. but I think it couldn't 
                    hurt to add a bit more paranoia to our tools before they flip the live bits.
16:36:24 < paulobanon_> were all problems in the tools been identified ?
16:36:30 < lmacken> even with all of the sanity checking bodhi could have done, it wouldn't have stopped 
                    what happened today with the comps..
16:36:42 < jima> how do we push bits out? rsync?
16:36:51 < lmacken> the repo that bodhi mashed was fine, but it synced out with a bad comps
16:36:57 < mmcgrath> lmacken: can't we add a check to bodhi does the sha1sum match whats in the xml file?
16:37:09 < lmacken> I haven't worked on mash at all..
16:37:14 < mmcgrath> <nod>
16:37:20 < lmacken> mmcgrath: yeah.. but that wouldn't have helped today, since everything matched up 
                    fine
16:37:35 < mmcgrath> jima: we just have one netapp in RDU where everything ends up.  It gets pushed from 
                     there to TPA and PHX, then everyone pulls from either TPA or PHX.
16:37:41 < mmcgrath> soon PHX will be the canonical location.
16:37:47 < jima> whatever the mechanism, i think we need to put in some kind of "speed bump" that 
                 detects if >x number of files get deleted.
16:37:53 < mmcgrath> s/soon/december sometime we think hopefully/
16:37:55 < notting> how did it get a 'bad' comps file? where did it come from?
16:38:01 < jima> if it exceeds the value, it requires human intervention.
16:38:08 < notting> jima: --max-delete?
16:38:08 < lmacken> all of the other disasters that you mentioned have hopefully been mitigated with 
                    some extensive sanity checking in bodhi
16:38:11 < mmcgrath> notting: we're not sure.
16:38:26 < mmcgrath> lmacken: do you have an ETA on the bodhi checks?
16:38:27 < jima> notting: !
16:38:39 < jima> notting: i wasn't even aware of that. wow.
16:38:51 < lmacken> notting: it didn't come from the repository that it was syncing out.. could have 
                    been the last one ?
16:38:59 < notting> lmacken: don't see how
16:39:01 < lmacken> mmcgrath: what checks ?
16:39:15 < mmcgrath> the "extensive sanity checking" :)
16:39:18 < lmacken> mmcgrath: already done
16:39:26 < jima> notting: alas, even that would probably kill everything after a few synchs.
16:39:27 < notting> lmacken: how does bodhi pass the comps file into mash?
16:39:28 < mmcgrath> oh ok.  so bodhi's good we think?
16:39:35 -!- clarkbw [n=clarkbw@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit "Ex-Chat"
16:39:43 < mmcgrath> and last nights issue was with mash, we're positive of that?
16:39:47 < lmacken> yep.. although doing some repodata sanity checking may not hurt
16:40:00 -!- SmootherFrOgZ [n=Smoother@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit 
          Read error: 113 (No route to host)
16:40:08 < lmacken> bodhi updates the comps module, and points mash at it.. so I'm assuming it was 
                    something with mash.
16:40:15 < mmcgrath> I suppose repodata checkign could at least produce an alert or something?  I'm not 
                     sure what the scripts should have done when they noticed a problem.
16:40:25 < notting> lmacken: all it does is take the file you pass in and pass it to createrepo
16:40:31 < lmacken> well, if there is a problem bodhi doesn't update the live symlink
16:41:34 < mmcgrath> lmacken: so bodhi calls on mash, mash messed up and then bodhi updated the live 
                     symlink?
16:41:40 < jima> mmcgrath: trigger a nagios warning? ;)
16:41:47 < mmcgrath> does mash have proper exit codes and such?
16:41:59 < lmacken> mmcgrath: no.. everything seemed to mash fine, and the repo looked good from bodhi's 
                    perspective.. so it updated the symlink
16:42:05 < lmacken> yeah, it does
16:42:14 < lmacken> bodhi rolls back when mash fails
16:42:23 < jima> (wait...that's actually not an awful idea.)
16:42:25 < notting> lmacken: was it a valid-but-old comps file? or....
16:42:57 < jima> mmcgrath: could the mash process output stderr to a file nagios could monitor for size?
16:43:08 < lmacken> notting: the checksums matched up from what I saw in the repo..
16:43:15 < jima> mash happens where? releng1?
16:43:20 < notting> lmacken: ?
16:43:32 < lmacken> jima: yes
16:43:40 < mmcgrath> jima: I think I'd rather monitor the mirror directly (checking sha1sums against 
                     whats there, making sure rpms are in the right dirs, that sort of thing.
16:43:48 < skvidal> no
16:43:50 < skvidal> notting: no
16:43:53 < skvidal> comps was corrupt
16:43:57 < jima> mmcgrath: 'k, just a thought.
16:44:07 < notting> skvidal: garbage? truncated? other?
16:44:14 < skvidal> garbage in a couple of lines
16:44:18 < skvidal> I have a ocpy of it
16:44:19 < skvidal> one sec
16:44:21 < mmcgrath> skvidal: do you happen to still have a copy?
16:44:21 < jima> nagios seems like a good mechanism for alerting people, though.
16:44:25  * mmcgrath never got a good look at it.
16:44:28 < skvidal> yes, one moment - putting it up
16:44:31 < mmcgrath> jima: it is.
16:44:45 < mmcgrath> jima: maybe for one of nagios's passive checks?
16:44:58  * mmcgrath admits to never having done passive checks in nagios.
16:45:01  * jima suddenly realizes he's probably the local nagios bitch now. ;)
16:45:01 < skvidal> line 80 has
16:45:02 < skvidal>    <descrات ذات ÙاجÙØ© رسÙÙ
ÙØ© Ù               إدارة حساؚات اÙ
     إدارة ا٠ا٠                       Ù
16:45:05 < skvidal> which is, umm, wrong :)
16:45:06 < jima> mmcgrath: me either.
16:45:12  * jima winces
16:45:21 < skvidal> http://skvidal.fedorapeople.org/misc/comps-f7.xml
16:45:22 < lmacken> skvidal: corruped on the mirrors.. but the past 4 repos that bodhi has mashed look 
                    fine..
16:45:27 < jima> skvidal: you bastard, you messed up my irssi session :)
16:45:50 < skvidal> jima: stop using old crap for your terminal then
16:46:07 < lmacken> it borked mine a bit.. screen maybe ?
16:46:08 < mmcgrath> skvidal: you and you're new fangled "GUI"
16:46:22 < jima> lmacken: irssi or screen, not sure.
16:46:33 < skvidal> mmcgrath: yah - I know - I'm a terrible person
16:46:45 < jima> skvidal: wah, wah, okay i'm over it.
16:46:46 < notting> skvidal: so, rsync or the netapp ate it?
16:46:48 < mmcgrath> so http://skvidal.fedorapeople.org/misc/comps-f7.xml compared to 
ftp://download.fedora.redhat.com/pub/fedora/linux/updates/7/i386/repodata/comps-f7.xml
16:46:50 < lmacken> skvidal: what is the sha1sum of that corrupted comps ?
16:48:10 < notting> yeah everything on the mash side looks sane
16:48:23 < lmacken> [bodhi@releng1 updates]$ find /mnt/koji/mash/updates/ -name 'comps-f7.xml' | xargs 
                    sha1sum | grep 1a140beb388014a7491fd359512fc9064d3eb1a4
16:48:26 < lmacken> nothing.
16:49:01 < skvidal> lmacken: 1a140beb388014a7491fd359512fc9064d3eb1a4  comps-f7.xml
16:49:12 < mmcgrath> 1a140beb388014a7491fd359512fc9064d3eb1a4 (bad)
16:49:17 < mmcgrath> 002c0212d55d200df7bdadcf83f3cb5c8d1a2114 (good)
16:49:45 < mmcgrath> skvidal: it is corrupt, xmlint does not like it.
16:49:48 < notting> so, we have a file corruption issue on the mirror master
16:49:53 < skvidal> mmcgrath: nod
16:49:55 < notting> um, 'yay'?
16:50:26 < jima> very yay.
16:51:05  * jima gets corrupt packages via rsync all the time :(
16:51:17 < mmcgrath> jima: define all the time
16:51:35 < jima> mmcgrath: once a week, maybe?
16:51:49  * jima uses mirrors.kernel.org, fwiw
16:52:04 < jima> not sure if i happen to be pulling files while they're still getting them, or what.
16:52:26 < jima> fedora/updates/7/i386/openoffice.org-langpack-nl-2.2.1-18.2.fc
16:52:28 < jima> 7.i386.rpm: (sha1) dsa sha1 MD5 GPG NOT OK
16:52:36 < jima> fedora/updates/7/i386/openoffice.org-langpack-tr_TR-2.2.1-18.2.fc7.i386.rpm: (sha1) dsa 
                 sha1 MD5 GPG NOT OK
16:52:46 < jima> fedora/updates/7/ppc/openoffice.org-langpack-pt_PT-2.2.1-18.2.fc7.ppc.rpm: (sha1) dsa 
                 sha1 MD5 GPG NOT OK
16:52:49 < jima> etc
16:52:52 < mmcgrath> notting: lmacken: so what do we do?
16:53:01 < mmcgrath> wait to see what happens tonight?  Try to recreate it?
16:53:02 < jima> (that was this morning, about 4 hours ago)
16:53:09 < jima> err
16:53:11 < jima> 12 hours
16:53:11 < notting> mmcgrath: wait and see if it happens again?
16:53:15 < mmcgrath> do we think the problem was on the file system or did createrepo bomb?
16:53:31 < notting> fs. the output of mash that is sent for rsync is fine
16:53:51 < lmacken> Once bodhi mashes the repos, what happens from there is black magic from my point of 
                    view.  Should we keep an eye on wallace or wherever snapmirror moves things to ?
16:54:17 < notting> lmacken: you *can*, but ... that's not supposed to eat itself. never has before :/
16:54:18 < mmcgrath> so wait, what was the sha1sum that mash outputted last night.
16:54:41 < notting> mmcgrath: mash outputted the good file. what ended up going on the mirrors was the 
                    bad file.
16:54:57 < lmacken> drwxrwxr-x 3 bodhi bodhi  4096 Oct  3 14:08 f7-updates-071003.1317
16:54:57 < lmacken> drwxr-xr-x 3 bodhi bodhi  4096 Oct  3 15:34 f7-updates-071003.1445
16:54:57 < lmacken> drwxr-xr-x 3 bodhi bodhi  4096 Oct  4 10:35 f7-updates-071004.0944
16:55:01 < lmacken> bodhi mashed those last night
16:55:06 -!- tibbs_ [n=tibbs@fedora/tibbs] has quit Read error: 104 (Connection reset by peer)
16:55:39 < warren> Does anyone have ssh access to wallace?
16:56:02 < notting> i do
16:56:06 < warren> We could put a cronjob on wallace to check for packages that fail integrity checks?
16:56:31 < mmcgrath> lmacken: whats the path to those files?
16:56:37 < lmacken> mmcgrath: /mnt/koji/mash/updates
16:57:58 < mmcgrath> skvidal: notting: can you guys remember the last time we shipped a 'corrupt' comps 
                     file?
16:58:08 < skvidal> no
16:58:30 < mmcgrath> k, we'll have to keep an eye on it.
16:58:32 < mmcgrath> paulobanon_: ping?
16:58:32 < skvidal> I can't remember comps eating it in the past
16:58:52 < mmcgrath> Ok, we'll have to keep an eye on it then.
16:58:57 < lmacken> warren: I think we should keep a close eye on it for now.. if it happens again, we 
                    may have bigger problems.
16:59:04 < mmcgrath> we're almost out of time.  I'm going to open up the floor to see if anyone has 
                     anything else to discuss
16:59:09 < paulobanon_> mmcgrath: 1/2 pong
16:59:11 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
16:59:45 < warren> mmcgrath, if there's nothing, please crash me again.
16:59:49 < mmcgrath> paulobanon_: will you be online in 8 - 10 hours or so?
17:00:19 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Role Call
17:00:19 < paulobanon_> ill be online at 07CET
17:00:22 < mmcgrath> Who's here?
17:00:28  * warren here
17:00:30  * paulobanon_ is here
17:00:31  * lmacken 
17:00:31 -!- jeremy [i=katzj@nat/redhat/x-8d6a13a6e0eccf5f] has quit Remote closed the connection
17:00:33 < mmcgrath> hehehe, nothing
17:00:36 < mmcgrath> there goes jeremy!
17:00:36 < skvidal> hahaha
17:00:38 < skvidal> jeremy!
17:00:42 < skvidal> lmacken: I think it's you
17:00:46 < skvidal> what char are you sending?
17:00:46 < lmacken> is it my blank /me ?!
17:00:51 < lmacken>  /me<space>
17:00:57 < lmacken> hahaha
17:01:00 < skvidal> I can't send that on here
17:01:01 < paulobanon_> lol
17:01:02 -!- warren [i=warren@nat/redhat/x-db77e9f6a3ecac4a] has quit Read error: 104 (Connection reset 
          by peer)
17:01:03  * mmcgrath 
17:01:06 -!- notting [i=notting@redhat/notting] has quit "Ex-Chat"
17:01:08 < paulobanon_> second one
17:01:08 < mmcgrath> that could  very well be.
17:01:12 < mmcgrath> oh they're dropping like flies
17:01:15 < skvidal> let's get jeremy back
17:01:20  * lmacken = l33t irc h4x0r
17:01:21 -!- warren [i=warren@nat/redhat/x-c58995287a57f383] has joined #fedora-meeting
17:01:24  * mmcgrath 
17:01:24 -!- warren [i=warren@nat/redhat/x-c58995287a57f383] has quit Remote closed the connection
17:01:26 < mmcgrath> sucker
17:01:28 < mmcgrath> hehehe
17:01:29 < skvidal> rock!
17:01:32 < lmacken> that's worse than WebTV's 40*'<' exploit
17:01:41 < mmcgrath> hilarious
17:01:43 -!- warren [i=warren@nat/redhat/x-55d0f9fd2fc5a883] has joined #fedora-meeting
17:01:49 < skvidal> do it again
17:01:50 < mmcgrath> warren: sorry, couldn't resist :)
17:02:00  * mmcgrath 
17:02:00 -!- warren [i=warren@nat/redhat/x-55d0f9fd2fc5a883] has quit Remote closed the connection
17:02:01 < abadger1999> mmcgrath: Way to hit him when's down :-)
17:02:07 < mmcgrath> skvidal told me to.
17:02:08  * lmacken \xeb\x1f\x5e\x89\x76\x08\x31\xc0
17:02:10 < lmacken> :(
17:02:15  * skvidal    \xeb
17:02:18 < mmcgrath> yeah, I think thats a confirmed.
17:02:20 -!- Affix [n=affix@xxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
17:02:35 < mmcgrath> Ok,
17:02:36 < skvidal> mmcgrath: let's test it on jeremy again, too
17:02:39 < mmcgrath> oh one more thing before I close the meeting.
17:02:40 -!- warren_temp [i=warren@nat/redhat/x-12661f6fa77b4b43] has joined #fedora-meeting
17:02:41 -!- warren_ [i=warren@nat/redhat/x-e2c822382a0d2309] has joined #fedora-meeting
17:02:42 < lmacken> hahah.
17:02:43 < warren_temp> non-proxy
17:02:45 -!- jeremy [i=katzj@nat/redhat/x-fb17ffc4fa607bbb] has joined #fedora-meeting
17:02:50 < skvidal> get them!
17:03:06 < mmcgrath> jima created this - http://publictest1.fedora.redhat.com/jima/maps/bittorrent/
17:03:07 -!- warren_ is now known as warren
17:03:09 < mmcgrath> it shows torrent seeds
17:03:10  * mmcgrath 
17:03:10 -!- warren [i=warren@nat/redhat/x-e2c822382a0d2309] has quit Remote closed the connection
17:03:10 -!- jeremy [i=katzj@nat/redhat/x-fb17ffc4fa607bbb] has quit Remote closed the connection
17:03:13 < mmcgrath> buhahaha
17:03:14 < skvidal> heee hee
17:03:18 < lmacken> :)
17:03:19 < skvidal> that rocks
17:03:19 < paulobanon_> lol
17:03:30 < warren_temp> who was it that time?
17:03:32 < skvidal> I mean that could almost be HOURS of fun
17:03:35 < skvidal> mmcgrath:
17:03:40 < jima> it's /me? wow.
17:03:41 < mmcgrath> warren_temp: that was me.  but this whole time its been lmacken's blank me
17:03:48 < lmacken> so good
17:03:50 -!- jeremy [i=katzj@nat/redhat/x-87c89897d9f154ea] has joined #fedora-meeting
17:03:51 < warren_temp> I don't even see blank /me's
17:03:59 < warren_temp> wow, that's so easy to exploit
17:03:59 < jima> i think i know what it is.
17:04:00 < mmcgrath> jeremy: sorry we'll stop now.
17:04:03 -!- jeremy [i=katzj@nat/redhat/x-87c89897d9f154ea] has left #fedora-meeting ["Ex-Chat"]
17:04:10  * lmacken will try and break the habit
17:04:12 < abadger1999> Hey ducked!
17:04:15 -!- jeremy [i=katzj@nat/redhat/x-87c89897d9f154ea] has joined #fedora-meeting
17:04:17 < abadger1999> s/Hey/He/
17:04:17 < f13> mmcgrath: pong
17:04:30 -!- frankc [i=824c405d@gateway/web/cgi-irc/ircatwork.com/x-3afc345cd68c3c7e] has left 
          #fedora-meeting []
17:04:32 < mmcgrath> jeremy: sorry, we figured out why you keep getting kicked off.
17:04:37 < skvidal> mmcgrath: I told him
17:04:50 < warren_temp> http://f3dora.org/paste/183  traceback
17:04:53 < mmcgrath> Oh hey, we've gone over time.
17:05:03 < mmcgrath> If no one says anything I'll close the meeting in 15 seconds.
17:05:08 < warren_temp> So... lmacken being absent
17:05:11 < mmcgrath> f13: no worries we were just discussing the comps thing.
17:05:13  * warren_temp shakes fist at lmacken 
17:05:17 -!- rdieter is now known as rdieter_away
17:05:19 < lmacken> warren_temp: sorry ! ;)
17:05:47 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Meeting End
17:05:51 < mmcgrath> Thanks for coming everyone.
17:05:53 < f13> mmcgrath: ok.
17:05:57 < f13> not my fault then?
17:06:09 < warren_temp> f13, notting wanted lspci on your powerpc mac mini
17:06:09 -!- mmcgrath changed the topic of #fedora-meeting to: Channel is used by various Fedora groups 
          and committees for their regular meetings | Note that meetings often get logged | For 
          questions about using Fedora please ask in #fedora | See 
          http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel for meeting schedule
17:06:28 < mmcgrath> f13: we never figured it out, mash produced the right output for comps, but that 
                     file didn't end up on the mirror.
17:06:52 < mmcgrath> a corrupt file did - http://skvidal.fedorapeople.org/misc/comps-f7.xml
17:06:59 -!- warren_temp [i=warren@nat/redhat/x-12661f6fa77b4b43] has left #fedora-meeting ["Leaving"]
17:07:00 -!- warren [i=warren@nat/redhat/x-578801bce06dfca2] has joined #fedora-meeting
17:07:28  * warren tests it on a different IRC network
17:07:44 < f13> mmcgrath: bizarro.
17:07:51 < f13> warren: were you able to get it?
17:07:58 < warren> f13, yes, booted a rescue CD
17:08:03 < mmcgrath> warren: want me to send a blank line?
17:08:06 < warren> mmcgrath, no
17:08:10 < mmcgrath> k
17:08:19 < warren> mmcgrath, I need the ability to do it to myself
17:08:23 < warren> lmacken, what client do you use?
17:08:25 < mmcgrath> k
17:08:28  * mmcgrath is on irssi
17:08:29 < warren> mmcgrath, so I can fix dircproxy
17:08:40 < mmcgrath> just a /me with a space after it
17:08:53 < warren> I wonder if this can be considered a remote DoS attack
17:08:54 < warren> =)
17:09:02 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has quit Read error: 110 (Connection timed 
          out)
17:09:29 < f13> warren: ok.
17:09:33 -!- lancelan__ [n=lance@xxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting
17:09:40  * mdomsch likes the FC6 ppc worldmap of torrent seeds
17:09:58 < lmacken> warren: irssi
17:10:55 < warren> cool
17:11:06 < warren> If you search google for "segfault" it also finds "segmentation fault"
17:11:41 < mmcgrath> mdomsch: heh, i didn't really look at that closely.
17:12:40 < ivazquez> Yeah, Google has gotten smrt.

Attachment: pgpxGC8rIEysu.pgp
Description: PGP signature