20:00 < mmcgrath> #startmeeting Infrastructure 20:00 < zodbot> Meeting started Thu May 27 20:00:42 2010 UTC. The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic: Infrastructure) 20:00 < mmcgrath> #topic Who's here? 20:00 -!- zodbot changed the topic of #fedora-meeting to: Who's here? (Meeting topic: Infrastructure) 20:00 * ricky 20:00 * nirik is lurking around. 20:00 < smooge> here 20:00 -!- sgallagh [~sgallagh@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:01 * abadger1999 here 20:01 * sgallagh lurking 20:02 < mmcgrath> Ok, lets get started 20:02 < mmcgrath> #topic Infrastructure Fedora 13 release 20:02 -!- zodbot changed the topic of #fedora-meeting to: Infrastructure Fedora 13 release (Meeting topic: Infrastructure) 20:03 < mmcgrath> #link https://fedorahosted.org/fedora-infrastructure/report/9 20:03 < mmcgrath> lets go through and make sure these are all closed 20:03 < mmcgrath> .ticket 2137 20:03 < zodbot> mmcgrath: #2137 (New website) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2137 20:03 < mmcgrath> ricky: mind closing that one? sijis has it currently 20:03 < ricky> Closed 20:03 < mmcgrath> .ticket 2138 20:03 < zodbot> mmcgrath: #2138 (Verify Mirror Space) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2138 20:04 < mmcgrath> smooge: mind closing that one? 20:04 < mmcgrath> .ticket 2139 20:04 < zodbot> mmcgrath: #2139 (Release day ticket.) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2139 20:04 < mmcgrath> this one's mine, closing 20:04 < smooge> done 20:04 < mmcgrath> .ticket 2141 20:04 < zodbot> mmcgrath: #2141 (Mirrormanager redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2141 20:04 < CodeBlock> here 20:04 < mmcgrath> mdomsch: that one all done? 20:04 * mmcgrath assumes so 20:04 < mmcgrath> .ticket 2146 20:04 < zodbot> mmcgrath: #2146 (Enable wiki caching.) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2146 20:04 < mmcgrath> .ticket 2147 20:04 < zodbot> mmcgrath: #2147 (Disable wiki caching) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2147 20:04 -!- egyDev [~egyDev@fedora/ahmed-araby] has quit Ping timeout: 258 seconds 20:04 < mmcgrath> smooge: those two done? 20:04 * mmcgrath assumes so :) 20:04 < mmcgrath> .ticket 2166 20:04 < zodbot> mmcgrath: #2166 (mirrors.fp.o/releases.txt) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2166 20:04 < smooge> yep. and should be closed 20:04 < mmcgrath> that one's done, closing now 20:05 < mmcgrath> And that leave just one more thing 20:05 < mmcgrath> .ticket 2145 20:05 < zodbot> mmcgrath: #2145 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2145 20:05 -!- jokajak [~jokajak@fedora/jokajak] has joined #fedora-meeting 20:05 < jokajak> sorry i'm late 20:05 < jokajak> i'm split between this and $dayjob 20:05 < mmcgrath> so all in all I think this release went well, we did launch pretty close to the target time. 20:05 < mmcgrath> jokajak: no worries. 20:05 < mmcgrath> ricky: how'd things go from your view with the website? 20:06 < ricky> Good, had to stay up the night before fixing up some docs stuff 20:06 < mmcgrath> yeah, I think docs was the one wildcard that wasn't great. 20:06 < ricky> Went smoothly on release day though, didn't delay release with the build for once :-) 20:06 < mmcgrath> but it didn't seem to impact users that much so not a big deal. 20:06 < mmcgrath> One thing we did see this time around that we had not seen in previous releases is http code 103. 20:06 < mdomsch> mmcgrath, yes 20:07 < mmcgrath> what is that you ask? It's not well defined but related to apache and caching. 20:07 < mmcgrath> It's not clear to me what the users were seeing when issued a 103 20:07 < mmcgrath> I do know it generally happened with larger(ish) content like images. 20:07 < mmcgrath> I couldn't recreate it. 20:07 < ricky> Images on fp.o? 20:07 < mmcgrath> ricky: yeah. 20:07 < ricky> Or do you mean ISO images? 20:07 < mmcgrath> pictures 20:07 < ricky> Yow, didn't know about that 20:07 < mmcgrath> it was reasonably rare compared to the number of served requests. 20:08 < mmcgrath> based on what I saw on the boxes, and from googling around my theory was that these were images in the process of loading that users either clicked the "stop" button killing the transfer. 20:08 < mmcgrath> or more likely, found the link they wanted before the whole page loaded. 20:08 < ricky> Ahh 20:09 < mmcgrath> but not being able to completely recreate it, it's hard to say. 20:09 < mmcgrath> anyone have any other questions or comments on this, the release, or anything? 20:09 < gholms> When I investigated a bunch of 103s happen for a while at work; that's what caused it. Recreating it is more reliable when you have an extremely short trip between a testing script and the server. 20:10 < mmcgrath> gholms: <nod> good to know. 20:10 < mmcgrath> I suspect it also helps if the server's under a lot of load and is slower than normal to load those things. 20:10 < gholms> I would try doing some testing from a machine in the datacenter if you can; it might be more reliably reproduced. 20:10 < mmcgrath> <nod> 20:11 < mmcgrath> Ok. 20:11 < mmcgrath> so next topic 20:11 * ricky will write a test script in a bit 20:11 < mmcgrath> #topic CDN 20:11 -!- zodbot changed the topic of #fedora-meeting to: CDN (Meeting topic: Infrastructure) 20:11 < mmcgrath> Thanks to nb I think our dnssec issues are re-fixed. 20:11 < mmcgrath> I think the next step is going to be geodns implementation. 20:11 < mmcgrath> which we've been testing on publictest8. 20:12 < mmcgrath> The CDN though is going to be a much more detailed project requiring quite a bit of our time to get in place and maintain. 20:12 < mmcgrath> but I think it will be worth it. 20:12 < mmcgrath> our end users should see much better performance then previously. 20:13 < mmcgrath> Anyone have any questions on this? 20:13 < mmcgrath> concerns? 20:13 < mmcgrath> want to help out? 20:13 < smooge> one sec 20:13 < smooge> CDN? 20:13 < ricky> Any idea what kind of technologies this will involve other than geodns? 20:13 < mmcgrath> content distribution network. 20:13 < mmcgrath> ricky: everything we have now 20:14 < mmcgrath> geodns was the last bit that'd make it worth while. 20:14 < mmcgrath> the work though is going to be making sure our caching layer is functioning properly. 20:14 < smooge> so an opensource akamai? 20:14 < ricky> Ah, cool, I'd definitely be happy to help out 20:14 < mmcgrath> smooge: yeah 20:14 < mmcgrath> the big thing is metrics. 20:14 -!- Oxf13 [~jkeating@fedora/ender] has quit Excess Flood 20:14 < mmcgrath> for example, we may want to look closely at serving static content from the proxy servers directly 20:14 < mmcgrath> like pkgdb's images and css. 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Changing host 20:15 -!- Oxf13 [~jkeating@fedora/ender] has joined #fedora-meeting 20:15 -!- Oxf13 [~jkeating@fedora/ender] has quit Excess Flood 20:15 < mmcgrath> we've had some issues in the past wrt caching our admin.fp.o content when someone is logged in. 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Changing host 20:15 -!- Oxf13 [~jkeating@fedora/ender] has joined #fedora-meeting 20:15 -!- Oxf13 [~jkeating@fedora/ender] has quit Excess Flood 20:15 < CodeBlock> I'd be happy to help out as well, just tell me what I'm doing 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:15 -!- Oxf13 [~jkeating@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Changing host 20:15 -!- Oxf13 [~jkeating@fedora/ender] has joined #fedora-meeting 20:16 -!- Ambassador [~Pk@unaffiliated/ambassador] has quit Ping timeout: 258 seconds 20:16 < mmcgrath> CodeBlock: k 20:16 < mmcgrath> does anyone have any experience actually setting these up? 20:17 < mdomsch> not for production use :-) 20:17 -!- Ambassador [~Pk@unaffiliated/ambassador] has joined #fedora-meeting 20:17 < mmcgrath> :) 20:17 < mmcgrath> well this will be an adventure for all of us. 20:17 -!- mcepl [~mcepl@xxxxxxxxxxxxxxxxxxxxxxxxxx] has left #fedora-meeting ["Odcházím"] 20:17 < mmcgrath> k, moving on if no one has anything else. 20:17 < smooge> mmcgrath, no my experience has been more in breaking them 20:17 < abadger1999> I think we've fixed the caching/logged in problem 20:17 < mmcgrath> smooge: :-D we'll need some of that too. 20:18 < abadger1999> (By setting no cookies on the particular content) 20:18 < mmcgrath> abadger1999: k, I may work with you on that to verify. Because when that happens... that is some scary crap. 20:18 < mmcgrath> :) 20:18 < abadger1999> <nod> 20:18 < mmcgrath> we have a decent staging environment now too so that should help. 20:18 < mmcgrath> For those that don't know what I'm talking about.... 20:18 -!- hydh [~www-data@fedora/sheid] has joined #fedora-meeting 20:18 < mmcgrath> when we first enabled caching on admin.fedoraproject.org. Login cookies were getting cached. So if toshio logged in before me. Then I tried to log in. It's possible i'd find myself logged in as toshio. 20:19 < mmcgrath> Boy was that a fun day. 20:19 < ricky> Hehe 20:19 < mmcgrath> ok, anyone have any other questions or comments on that? 20:19 < skvidal> I wanna be toshio! 20:19 < mmcgrath> ok, that transitions into 20:19 < mmcgrath> #topic Internetx -- new sponsor 20:19 -!- zodbot changed the topic of #fedora-meeting to: Internetx -- new sponsor (Meeting topic: Infrastructure) 20:19 < mmcgrath> I'm happy to say we have a new machine in the EU (hydh hooked us up) 20:20 < skvidal> so are they all ipv6 all the time? 20:20 < mmcgrath> it has ipv6, good connection. 20:20 < skvidal> if so then it's nice to see puppet (and func) both work sanely 20:20 < mmcgrath> they have both ipv6 and ipv4. 20:20 < hydh> native ipv6 20:20 < smooge> skvidal, I tried to be toshio but my feet grew 3 feet 20:20 < hydh> and 1gbit uplink 20:20 < mmcgrath> I'm thinking at a minimum we're going to need to move noc2 out there. 20:20 < ricky> Yeees! 20:20 < smooge> yes 20:20 < mmcgrath> because with 2 ipv6 connections, it's time to start actually monitoring them. 20:21 < mmcgrath> and I'm honestly not sure if nagios supports it. 20:21 < mmcgrath> though I'd imagine it does. 20:21 < smooge> nagios 3 does.. [isn't that the default line for anything that nagios currently doesnt?] 20:21 < mmcgrath> hahahahah 20:21 < mmcgrath> does nagios bake cookies? 20:21 < mmcgrath> nagios 3 does... 20:21 < smooge> mmmmmm coookies 20:22 * gholms has a nagios plugin alerting him to lunch time as it approaches 20:22 < skvidal> mmcgrath: if you combine nagios 3 with butrfs it cures cancer! 20:22 < mmcgrath> Ok, so we'll be getting that brought online soon. 20:22 < rsc> mmcgrath: InterNetX from Germany is sponsoring Fedora? 20:22 < mmcgrath> smooge: Ponies! 20:22 < mmcgrath> rsc: yup 20:22 < mmcgrath> rsc: so we can finally close some of your "X is slow to load" tickets :) 20:22 < mmcgrath> ricky: how's proxy2 going btw? 20:22 < ricky> So far so good, done with puppet, getting it on func 20:23 < rsc> mmcgrath: that would be great. Because InterNetX has powerful infrastructure and even well IPv6 connectivity :) 20:23 < mmcgrath> ricky: ever figure out what was causing that error? 20:23 < ricky> Nope, but I'm definitely not done looking :-) 20:23 < mmcgrath> rsc: it's a done deal. It's already handed over to us, we're just still in the process of building it :) 20:23 -!- bpepple [~bpepple|l@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit Quit: Ex-Chat 20:23 < smooge> ponies and cookies. can my day get any better 20:23 < mmcgrath> ricky: did you just end up using my.. eh hem... hack? 20:23 < ricky> Yeah 20:23 < ricky> Shocking how fast it still was.. 20:25 < mmcgrath> yeah 20:25 < mmcgrath> anyone have anything else on this? 20:25 < smooge> not me 20:25 < mmcgrath> ok. next one 20:25 < mmcgrath> #topic koji backups 20:25 -!- zodbot changed the topic of #fedora-meeting to: koji backups (Meeting topic: Infrastructure) 20:25 < mmcgrath> so I've been working to move koji backups from the tape drives to dedicated storage. 20:25 < mmcgrath> in this case 6U of storage including a 2U server and 2 2U disk trays. 20:25 < skvidal> ricky: getting it on func? doesn't puppet do that automagically now? 20:26 < skvidal> ricky: it's the same cert 20:26 < ricky> skvidal: Thank you - you just saved me the time I was going spend debugging nothing :-) 20:26 < mmcgrath> It's been backing up for about a week and a half now. 20:26 < skvidal> ricky: if you login to puppet1 20:26 < skvidal> you can run 20:26 < smooge> is it because of disk speed OR just churn of whats in koji? 20:27 < skvidal> sudo func 'proxy02*' call test ping 20:27 < skvidal> ricky: which will tell you if it works 20:27 < mmcgrath> smooge: well it doesn't seem to be the disks on the backup server. Must just be the /mnt/koji speed. 20:27 < skvidal> damn it 20:27 < skvidal> I need to write up the docs on using func in FI 20:27 * skvidal makes a note 20:27 < smooge> skvidal, the sticky note from last week fell off? 20:27 < mmcgrath> anyone have any questions or comments on that? 20:28 -!- XulLunch is now known as XulWork 20:28 < mmcgrath> k 20:28 < mmcgrath> next topic 20:28 < mmcgrath> #topic /mnt/koji 20:28 -!- zodbot changed the topic of #fedora-meeting to: /mnt/koji (Meeting topic: Infrastructure) 20:28 -!- ldimaggi_ [~ldimaggi@nat/redhat/x-jdqzmxhsvlfgnilp] has quit Quit: Leaving 20:28 < mmcgrath> from backups, onto the real deal. 20:28 < skvidal> smooge: be nice 20:28 < skvidal> :) 20:28 < mmcgrath> dgilmore: when do you want to move /mnt/koji over? 20:28 < dgilmore> mmcgrath: it took me over a week to rsync /mnt/koji/packages onto the equalogix 20:28 < mmcgrath> I assume we're going to want to wait until the other equalogicx is on site? 20:28 < mmcgrath> mdomsch: you know we own two of those now right? 20:28 < dgilmore> mmcgrath: need to do a full sync again 20:29 < dgilmore> mmcgrath: and i want to test my rm -rf 20:29 < mdomsch> mmcgrath, no, that's great! 20:29 < dgilmore> that will free up 1.2T or so 20:29 < mmcgrath> mdomsch: they're not both on site yet but they will be. 20:29 < dgilmore> mmcgrath: and id kinda like to wait till we get the second unit in place 20:29 < mmcgrath> dgilmore: I have some port concerns. 20:29 * CodeBlock needs to head out for a while, sorry. back later 20:29 < mmcgrath> dgilmore: makes me wonder if we can use crossover cables. 20:29 < mmcgrath> CodeBlock: no worries. 20:29 < dgilmore> mmcgrath: how we are using it we will only use a single port ever 20:30 < dgilmore> mmcgrath: it doesnt do port bonding 20:30 < mmcgrath> dgilmore: k, we may need to communicate that to the network team because last time I was there I swear we had like 6 ports plugged in. 20:30 < dgilmore> mmcgrath: with a single client we are really only using a single port 20:30 < mmcgrath> oh that's right, or it does do bonding, just not the type our network team can work with. 20:30 < mmcgrath> yeah. 20:30 < dgilmore> it doesnt do bonding period 20:31 < mmcgrath> dgilmore: it'd be nice to have the two units talking to eachother over a dedicated link though 20:31 < dgilmore> its designed to balnace client load by sending different clients to different ports 20:31 < mmcgrath> or is that not how it's to be setup? 20:31 < dgilmore> mmcgrath: not really how its designed 20:31 < mmcgrath> k, well when the time comes I'll leave it to you. 20:32 < mmcgrath> I would like to have this all up and running asap. only because it takes so long to get going, I'd hate for this to bump up into the alpha. 20:32 < dgilmore> our useage is really outside of there normal use case 20:32 < mmcgrath> dgilmore: do you know when the equalogic will ship? 20:32 -!- fbijlsma [~fbijlsma@xxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 20:32 < dgilmore> mmcgrath: let me ask my boss 20:32 < smooge> when will we get it out there? and do you need me to be there physically? 20:33 < dgilmore> smooge: someone will need to rack it 20:33 < mmcgrath> smooge: I don't think we will, jonathan setup the last one. I suspect he will this time too. 20:33 < dgilmore> and hook up serial port 20:33 -!- kc8hfi [~kc8hfi@xxxxxxxxxxxxxx] has quit Quit: home! 20:33 < smooge> okit dokit 20:33 < dgilmore> but i hope johnathan can do it 20:34 < mmcgrath> k, any other questions on that? 20:34 < dgilmore> with the second unit i want to do raid10 over the 32 1tb drives in the 2 units 20:34 < mmcgrath> if not we'll move on. 20:34 < dgilmore> but im done now 20:35 < mmcgrath> alrighty 20:35 < mmcgrath> with that 20:35 < mmcgrath> #topic open floor 20:35 -!- zodbot changed the topic of #fedora-meeting to: open floor (Meeting topic: Infrastructure) 20:35 < mmcgrath> anyone have anything they'd like to discuss? 20:35 < mdomsch> mmcgrath, did you ever #endmeeting the meetbot in -admin ? 20:35 * ricky checked this morning and it was gone 20:35 < mdomsch> ok 20:35 < mmcgrath> mdomsch: I did :) 20:36 < mmcgrath> towards the end of the day after traffic was nearing back down to normal I ended it. 20:36 < mmcgrath> I should have sent the logs to the list, but they are available 20:36 < mdomsch> np 20:36 < mdomsch> it wasn't a very exciting day - just like I like it 20:36 -!- mchua is now known as mchua_afk 20:36 < mdomsch> either our processes have gotten so good that release day is a non-event 20:36 < mdomsch> or our traffic was down, or both 20:37 < mmcgrath> no doubt. 20:37 * mmcgrath thinks it was a little of both. 20:37 < mdomsch> thanks to Oxf13 for getting the bits to the mirrors so early 20:37 < mmcgrath> but if things aren't broke, we're doing what we can :) 20:37 < mmcgrath> yeah, that helps too I bet quite a bit 20:37 < mdomsch> I didn't hear anyone complain about not getting the bits; I did hear about slow torrents 20:37 < mmcgrath> when people show up from the release and can't get to it, I suspect there's a lot more re-loading on our servers. 20:37 < mdomsch> but that's also because they aren't advertised so much anymore 20:37 < mmcgrath> mdomsch: I had good luck with the torrents but I wasn't paying attention to all of them. 20:38 < smooge> is so used to having torrents blocked that I forgot to do anything with them 20:38 < mmcgrath> :) 20:38 < mmcgrath> ok, anyone have anything else to discuss? 20:38 < smooge> not me 20:38 < mdomsch> all quiet here 20:39 < smooge> going to just shoot the telemarketers who keep calling 20:39 < mmcgrath> alrighty, we'll close in 30 20:39 -!- MatthewCascio [~MatthewCa@xxxxxxxxxxxxx] has joined #fedora-meeting 20:39 < dgilmore> smooge: ive had a bunch of them recently 20:39 < mmcgrath> :) 20:39 < mmcgrath> ok 20:39 < smooge> election people wanting me to vote for someone.. but I unregistered from parties last year so they are wasting their time 20:39 < mmcgrath> #endmeeting 20:39 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule 20:39 < zodbot> Meeting ended Thu May 27 20:39:50 2010 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . 20:39 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2010-05-27/fedora-meeting.2010-05-27-20.00.html 20:39 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-05-27/fedora-meeting.2010-05-27-20.00.txt 20:39 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2010-05-27/fedora-meeting.2010-05-27-20.00.log.html
Attachment:
pgpK30AMt46hw.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
