15:55 * jima runs and hides 15:58 -!- fchiulli [i=824c4010@gateway/web/cgi-irc/ircatwork.com/x-a6b81cd860eb6651] has joined #fedora-meeting 16:00 < jima> AIEEE NOOO 16:00 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Who's here? 16:00 < mmcgrath> Ping 16:00 < mmcgrath> who's around? 16:01 < markhsa> Mark H 16:01 < paulobanon_> me 16:01 < markhsa> me 16:01 < f13> I'm here 16:01 < fchiulli> fchiulli is listening 16:01 * jeremy is around-ish 16:01 * quaid is lurking 16:01 < bklemm> bklemm is listening in too 16:01 * jima imagines fchiulli and bklemm are new to irc :) 16:02 * jima 16:02 < mmcgrath> spevack: abadger1999 poelcat mbacovsk ricky ivazquez warren skvidal lmacken dgilmore: ping 16:02 < fchiulli> Give that man a cigar. 16:02 < mmcgrath> I know some of them won't be in today. 16:02 < skvidal> pong 16:02 < abadger1999> pong 16:02 < warren> pong 16:02 < poelcat> mmcgrath: pong 16:02 < jima> dgilmore was busy as heck earlier, not sure about now. 16:03 -!- giarc_w [i=hidden-u@xxxxxxxxxxxxxxxx] has joined #fedora-meeting 16:03 < lmacken> pong 16:03 < mmcgrath> Welp, lets get started then. 16:03 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Release is go! 16:03 < mmcgrath> .tiny https://hosted.fedoraproject.org/projects/fedora-infrastructure/query?status=new&status=assigned&status=reopened&milestone=Fedora+8 16:03 < mmcgrathbot> mmcgrath: http://tinyurl.com/292qq8 16:04 < mmcgrath> So yeah, AFAIK we will release Fedora 8 on the 8th next week. 16:04 < mmcgrath> Is anyone confused about what they should or shouldn't be doing at this point? 16:04 < jima> should: looking for bugs 16:04 < spoleeba> mmcgrath, duck and cover? 16:04 < warren> meh 16:04 < jima> shouldn't: changing infrastructure without asking :) 16:05 < skvidal> jima: wait? what? 16:05 < skvidal> should NOT 16:05 < jima> spoleeba: i thought we did that every day ;) 16:05 < skvidal> oh shit 16:05 < abadger1999> :-) 16:05 * mmcgrath smacks skvidal 16:05 -!- nim-nim [n=nim-nim@fedora/nim-nim] has quit "Leaving." 16:05 < skvidal> mmcgrath: hey, I wasn't the one who brought proxy4 online yesterday! :) 16:05 < mmcgrath> Just a reminder, ANOTHER one :) After tonight, no changes to infrastructure without contacting the list first. 16:05 < jima> skvidal: ...? 16:05 < skvidal> jima: I'm making fun of mmcgrath 16:06 < mmcgrath> jesse also had a wish to extend this period for next release. I'm generally in favor of that though I'd like to see hwo this change freeze goes. 16:06 * dgilmore is sitting on the cold floor of the colo 16:06 < mmcgrath> There are a few critical things I'd like to talk about. 16:06 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure - mmcgrath (schedule) 16:06 < jima> hmm, should i be getting approval to work on ticket #220? :) 16:06 < mmcgrath> jima: lets get to that in a bit 16:06 < jima> 'k 16:06 -!- nim-nim [n=nim-nim@fedora/nim-nim] has joined #fedora-meeting 16:07 < mmcgrath> So, if I've done my job right, I won't need to be around for this release. Having said that. I'm going to try to be. 16:07 < paulobanon_> no changes no fun! 16:07 < mmcgrath> skvidal is going to be the point person during the release, even if I am around. 16:07 < skvidal> mmcgrath: do not be around 16:07 < skvidal> seriously - don't do it 16:07 < mmcgrath> :) 16:07 < jima> skvidal: see, you shouldn't have made fun of him -- he just got payback. 16:07 < skvidal> we'll deal w/o you :) 16:07 < mmcgrath> So here's my schedule. 16:07 < skvidal> jima: I [was] volunteered 16:08 < mmcgrath> I'm off tomorrow and will largely be unavailable until Monday. 16:08 < jima> skvidal: exactly my point. ;) 16:08 < jima> mmcgrath: have fun! :) 16:08 < mmcgrath> I'll be taking half days Monday - Thursday (release day) and on the 9th I'll be gone. 16:08 < mmcgrath> just gone. Like I wasn't ever here. 16:08 < jima> damn ninja. 16:08 < mmcgrath> and I'll be back for good on the 13th. 16:08 < jeremy> who's mmcgrath? 16:08 < skvidal> mmcgrath: wow, like keyser sose 16:08 < mmcgrath> Hopefully with a tan. Though its likely I'll burst into flames the first time I come in contact with the sun. 16:09 < skvidal> KEYSER SOSE KEYSER SOSE! 16:09 < mmcgrath> :) 16:09 < jima> i think anything bursts into flame when it comes even *close* to the sun. 16:09 < mmcgrath> So mark it down and remember that we talked about it in the meeting. 16:09 < mmcgrath> Next topic! 16:09 < paulobanon_> wedding topic ?! 16:09 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Current nown oddities. 16:09 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Current known oddities. 16:09 < mmcgrath> damn 16:09 < mmcgrath> Ok. 16:09 < jima> known oddity: "nown" is missing a 'k' 16:09 < skvidal> so, next week after we remove all of mmcgrath's accounts what will we be doing 16:09 < skvidal> ? 16:09 < skvidal> oh 16:09 < skvidal> damn 16:10 < skvidal> wrong channel 16:10 < jima> skvidal: that was for #fedora-evil, huh? 16:10 < mmcgrath> heh 16:10 < skvidal> it was 16:10 < mmcgrath> https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/222 16:10 < mmcgrath> .ticket 222 16:10 < mmcgrathbot> mmcgrath: #222 (sysctl on the proxy servers) - Fedora Infrastructure - Trac 16:10 < mmcgrath> I should have had this done a week ago but testing it and being busy with other stuff put it off. 16:10 < mmcgrath> Long story short. 16:10 -!- stahnma_ [n=stahnma@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-meeting 16:10 < yingbull> I should probably say hi - I was around earlier in the fall, but got a bad case of the $DAYJOB, and had fedora account issues that slowed me down getting involved (and without anything tasked to me yet, I just sorta faded off). Back around now, tho. So... hi. ;-) 16:10 < mmcgrath> The sysctl configuration optimizations have been done on all the proxy servers. 16:10 < mmcgrath> yingbull: welcome! 16:11 < yingbull> made it to this meeting at least, hopefully its a good start. :-) Sorry to interrupt. 16:11 < mmcgrath> So here's the scoop, we've got optimized proxy servers. I've done all the testing I can to ensure that we are getting peak performance from them. 16:11 * jima waves to yingbull 16:11 < jima> mmcgrath: seems like a clever idea, yeah. 16:11 < jima> is that all the proxies or just 1-2? 16:11 < mmcgrath> Having said that, I don't feel they're tested enough to just deploy. So to mitigate this risk my *optimized* config is in /root/sysctl.conf and the normal config thats been deployed for many months is in /etc/sysctl.conf 16:11 < mmcgrath> jima: all of them. 16:12 < mmcgrath> So if anyone sees the proxy servers getting overwhelmed, and feel it might be tuned wrong, just run "sysctl -p" 16:12 < mmcgrath> if you want to go back to the 'optimized configs' run "sysctl -p /root/sysctl.conf" 16:12 < mmcgrath> profit. 16:12 * jima almosts asks "where's a copy of the optimized one in puppet?" then remembers mmcgrath provided us with a copy. 16:12 < mmcgrath> Thats an odditiy I thought I'd mention. 16:12 -!- clarkbw [i=clarkbw@nat/redhat/x-ae3597e3eeab15df] has joined #fedora-meeting 16:12 < mmcgrath> since it doesn't make any obvious sense. 16:13 -!- londo [n=georgiou@xxxxxxxxxxxx] has quit Read error: 104 (Connection reset by peer) 16:13 -!- clarkbw [i=clarkbw@nat/redhat/x-ae3597e3eeab15df] has quit Read error: 104 (Connection reset by peer) 16:13 < mmcgrath> Anyone have any questions on that? Did anyone see anything obviously wrong with the sysctl.conf I sent to the list? 16:13 < lmacken> looked sane to me 16:13 < jeremy> seem fine to me 16:13 < mmcgrath> ok, moving on. 16:14 < mmcgrath> Oddity number 2. 16:14 -!- rdieter is now known as rdieter_away 16:14 < mmcgrath> The wiki notifications today. 16:14 < mmcgrath> We've been seeing higher than normal wiki notifications as of today. 16:14 < mmcgrath> I don't think the wiki is any worse off today then it was yesterday. I think its because we've been monitoring the wiki with caching enabled and I recently made that cache more aggressive than normal. 16:15 < mmcgrath> The notifications we're getting are from the only non-cached page. 16:15 < mmcgrath> Which means A) the wiki is unresponsive at times. 16:15 < mmcgrath> but B) I've not witnessed this myself, only seen the nagios alerts. 16:16 < mmcgrath> I'm watching this closely and trying to tune the wiki a bit but so far nothing has made the load any better. 16:16 < mmcgrath> Long story short, the caching I added sees between a 50% and 75% hit rate. 16:16 < jima> that's good. 16:16 < mmcgrath> that means only 50% - 25% of the requests actually hit the app server. 16:16 < mmcgrath> this is something we did not have for the F7 release. 16:16 -!- clarkbw [i=clarkbw@nat/redhat/x-29205887cfe9d537] has joined #fedora-meeting 16:16 < mmcgrath> The bulk of our work will be done by the proxy servers this time around. 16:17 < jima> but god help the actual wiki for the hits that do get through. :| 16:17 < mmcgrath> of which we have 4, the 2 new ones have more ram and are x86_64 boxes. So I think we're in good shape. 16:17 < mmcgrath> The wiki, as always, concerns me. 16:17 -!- k0k [n=k0k@fedora/k0k] has quit Read error: 110 (Connection timed out) 16:17 < mmcgrath> It just does. 16:17 * jima concurs 16:17 < mmcgrath> I've moved app3 off of xen1 and on to xen2 to dedicate more resources to xen1 (which currently contains the wiki and one proxy server) 16:17 < jeremy> we should probably just be aggressive about making "important" pages static and for the rest, if the wiki goes down, the wiki goes down 16:17 < poelcat> skvidal: is the problems I had yesterday app1 and with updating wiki pages is fixed? 16:17 < mmcgrath> unfortunately thats the best setup I've found so far. 16:18 < skvidal> poelcat: for certain values of fixed, yes 16:18 < mmcgrath> Any questions about the wiki right now? 16:19 < jima> mmcgrath: yes, but my questions keep timing out. 16:19 < mmcgrath> I've also turned off the translate app for app1 for now. 16:19 * mmcgrath smacks jima. 16:19 < mmcgrath> especially over the next many days. If you have ANY problems with the wiki, please email me the time, what you were trying to do and your IP address. 16:20 < mmcgrath> Ok, moving on. 16:20 < mmcgrath> mdomsch is not around but all that is left on that end is to disable the primary mirrors initially (download.fedora.redhat.com) from the mirror list and public list. 16:20 * jima scratches head 16:21 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Specific tickets open. 16:21 < jima> disable them from mirrorlist? isn't that tweakable via mirrormanager? 16:21 < mmcgrath> .ticket 93 16:21 < mmcgrathbot> mmcgrath: #93 (Fedora 8) - Fedora Infrastructure - Trac 16:21 < mmcgrath> We'll close that when everything is done 16:21 < mmcgrath> .ticket 97 16:21 < mmcgrathbot> mmcgrath: #97 (Fedora 8 Website) - Fedora Infrastructure - Trac 16:21 < mmcgrath> Ricky and I have been working on this together, expect to see the F8 website by the end of the day but with the F7 graphics and such. 16:21 < mmcgrath> We'll have a special build for F8 for the mirrors by the end of the day on Monday. 16:21 < mmcgrath> .ticket 99 16:21 < mmcgrathbot> mmcgrath: #99 (Mirror Coordination) - Fedora Infrastructure - Trac 16:22 < mmcgrath> Currently assigned to mdomsch either he or jesse keating will likely notify the mirrors when the bits are ready. 16:22 < mmcgrath> .ticket 100 16:22 < mmcgrathbot> mmcgrath: #100 (Web Content on the mirrors) - Fedora Infrastructure - Trac 16:22 < mmcgrath> already discussed, this will be done with ticket #97 16:22 < mmcgrath> .ticket 101 16:22 < mmcgrathbot> mmcgrath: #101 (Fedora Infrastructure Change Freeze) - Fedora Infrastructure - Trac 16:22 < mmcgrath> I'll close this ticket AFTER tonight. 16:22 < mmcgrath> ticket .137 16:22 < mmcgrath> .ticket 137 16:22 < mmcgrath> :) 16:22 < mmcgrathbot> mmcgrath: #137 (Sponsorship prominence) - Fedora Infrastructure - Trac 16:23 < mmcgrath> This is actually in good shape and live. Its linked at the bottom of the page http://fedoraproject.org/sponsors 16:23 < mmcgrath> .ticket 210 16:23 < mmcgrathbot> mmcgrath: #210 (Disable F8 in Mirrormanager) - Fedora Infrastructure - Trac 16:23 < notting> g 16:24 < mmcgrath> This is also currently assigned to mdomsch (who's not here) but I'll follow up with him to make sure this is done correctly when the bits go 16:24 < mmcgrath> .ticket 211 16:24 < mmcgrathbot> mmcgrath: #211 (Enable Bodhi Updates) - Fedora Infrastructure - Trac 16:24 < mmcgrath> lmacken: ? 16:24 < mmcgrath> This is assigned to you, still all good? 16:24 < lmacken> shouldn't be a problem 16:24 < mmcgrath> any ETA? 16:25 < lmacken> it's just a bitflip.. whenever we want to unleash 16:25 -!- stahnma_ is now known as stahnma 16:25 < mmcgrath> k, I'll leave that up to you and releng then :) 16:25 < mmcgrath> .ticket 54 16:25 < mmcgrathbot> mmcgrath: #54 (Postfix Server) - Fedora Infrastructure - Trac 16:25 < mmcgrath> This, unfortunately, will probably be the only ticket we didn't completely finish. Mostly an -ENOTIME and a -EALREADYHAVESOMETHING 16:26 < mmcgrath> bastion's been fine, this was a bigger priority when the mailman list stuff was being discussed and we had OTRS. 16:26 < mmcgrath> and the last ticket 16:26 < mmcgrath> .ticket 136 16:26 < mmcgrathbot> mmcgrath: #136 (Translated Website) - Fedora Infrastructure - Trac 16:26 < mmcgrath> We've 'frozen' the website and the translation team is currently at work on it 16:26 < jima> yay for translators! 16:26 * mmcgrath notes 16:26 < mmcgrath> http://translate.fedoraproject.org/module/fedora-web 16:27 < mmcgrath> I think this is actually a big deal. 16:27 < mmcgrath> but the final 'freeze' will be on monday night. 16:27 < mmcgrath> So really, we're in good shape. 16:27 < mmcgrath> Hopefully we won't have to juggle much at release time. 16:27 < mmcgrath> Anyone have any questions or concerns regarding me, our infrastructure, stuff not staying up or getting overloaded? 16:28 < mmcgrath> skvidal: you've got to have questions about stuff right? 16:29 < skvidal> mostly i'm reading 16:29 -!- EvilBob [n=bob@fedora/pdpc.sustaining.BobJensen] has quit Remote closed the connection 16:29 * mmcgrath also notes http://fedoraproject.org/wiki/Infrastructure/SOP/Release 16:29 < mmcgrath> still exists and is valid. 16:29 < skvidal> but right now my primary thoughts are making sure we have all the steps down for dynamically making static pages when we need them 16:29 < mmcgrath> skvidal: we should make an SOP for that. 16:30 < mmcgrath> skvidal: OH, thats actually a good point. 16:30 < skvidal> but we can't get to the SOP :) 16:30 < skvidal> so it's better just to have a note 16:30 < skvidal> so i'm thinking 16:30 < mmcgrath> *if* the wiki gets overloaded and you can't get to it, log into RH VPN and go to app2.fedora.phx.redhat.com/wiki/ 16:30 < mmcgrath> since we can't balance the wiki its always on one or the other, just go to the other :) 16:30 < skvidal> 1. save the page in firefox 16:30 < mmcgrath> skvidal: do it with wget. 16:30 < skvidal> 2. upload the files to the static path on the proxies 16:31 < skvidal> wget, in the past, didn't grab css for me 16:31 < mmcgrath> hmm, k, we can test that after the meeting. 16:31 < skvidal> okie doke 16:31 < mmcgrath> I'd also encourage everyone to become familiar with our caching setup. especially the headers. 16:32 < mmcgrath> .headers http://fedoraproject.org/wiki/ 16:32 < mmcgrathbot> mmcgrath: apptime: D=77335, proxytime: D=167438, expires: Thu, 01 Nov 2007 21:32:01 GMT, vary: Cookie,User-Agent,Accept-Language, connection: close, server: Apache/2.2.3 (Red Hat), appserver: (null), proxyserver: proxy3.fedoraproject.org, date: Thu, 01 Nov 2007 20:32:01 GMT, content-type: text/html; charset=utf-8 16:32 * mmcgrath notes proxy and app time 16:32 < mmcgrath> .headers http://fedoraproject.org/wiki/ 16:32 < mmcgrathbot> mmcgrath: apptime: D=77335, content-length: 13644, proxytime: D=87, expires: Thu, 01 Nov 2007 21:32:01 GMT, vary: Cookie,User-Agent,Accept-Language, connection: close, server: Apache/2.2.3 (Red Hat), appserver: (null), proxyserver: (null), date: Thu, 01 Nov 2007 20:32:15 GMT, content-type: text/html; charset=utf-8, age: 15 16:32 * mmcgrath notes "age" this time. 16:32 < mmcgrath> stuff like that is important for understanding why USER A) gets cached copy and why user B) doesn't. 16:33 < mmcgrath> If you want to flush all the cache, just "rm -rf /srv/web/cache/*" on the proxy box. 16:33 -!- londo [n=georgiou@xxxxxxxxxxxx] has joined #fedora-meeting 16:33 < mmcgrath> blah blah. 16:33 < mmcgrath> Ok, if no one has any questions about anything else I'll open the floor? 16:33 < skvidal> mmcgrath: one more question 16:33 < skvidal> on the proxy boxes 16:33 < skvidal> we have /srv/web/static-web 16:34 < skvidal> is that path old/broken? 16:34 < mmcgrath> That is the old path. 16:34 * mmcgrath notes proxy4 16:34 < mmcgrath> it doesn't have it. 16:34 < skvidal> right - I was more thinking of cleaning that up 16:34 < mmcgrath> <nod> 16:34 < mmcgrath> I thought I'd moved that to /tmp/ 16:35 < mmcgrath> perhaps I moved it back during the conversion. 16:35 < mmcgrath> but yeah, that should now be "/srv/web/fedoraproject.org/" or "/srv/web/start.fedoraproject.org/" 16:35 < skvidal> looks like you copied it to temp :) 16:35 < skvidal> but it never got nuked 16:35 * mmcgrath also notes that AFAIK, search.fedoraproejct.org won't be ready. 16:36 < mmcgrath> but thats not really our concern, just pointing it out. 16:36 < yingbull> my bit: I'll want something to do soon. I need to fix my fedora account, but will want something to work on to get started. Else I'm likely to let $DAYJOB eat my $NIGHTLIFE, and without anything depending/waiting on me, fade away again. :-) Anyone got something that I can thing them about once my fedora account is working? 16:36 < mmcgrath> yingbull: how's your mysql skills? 16:36 -!- clarkbw [i=clarkbw@nat/redhat/x-29205887cfe9d537] has quit Read error: 110 (Connection timed out) 16:37 < yingbull> mmcgrath: not bad, haven't had a huge need for it lately, but I did work with it before. 16:37 < yingbull> Used it for a big web-app at the last place I worked. 16:37 < mmcgrath> yingbull: ping me after the meeting in #fedora-admin, we'll find something definitive for you. 16:37 < yingbull> Will do. Save me from $DAYJOB, please! :-) 16:37 < mmcgrath> heh 16:37 < mmcgrath> ok 16:37 -!- mmcgrath changed the topic of #fedora-meeting to: Infrastructure -- Open Floor 16:38 < mmcgrath> Does anyone have anything specific they'd like to discuss? 16:38 < mmcgrath> I'd encourage everyone to start thinking of F9 goals. 16:38 * jima muses 16:39 < mmcgrath> Anyone have anything else? If not we'll close the meeting. 16:39 < jima> mmcgrath: ticket 220? 16:40 < mmcgrath> .ticket 220 16:40 < mmcgrathbot> mmcgrath: #220 (nagios monitoring for puppet1) - Fedora Infrastructure - Trac 16:40 < mmcgrath> jima: ahh, yeah I just wanted to make sure it was beign monitored. 16:40 < jima> it's not. next question? :) 16:41 < mmcgrath> well, can we make it monitored? ICMP, disk, puppetmaster, http. 16:41 < mmcgrath> standard stuff. 16:41 < jima> is it running nrpe? 16:41 < mmcgrath> it should be. 16:41 < jima> yep, it is. 16:42 < jima> should i make those changes to nagios? no conflicts with the infrastructure freeze? 16:42 < mmcgrath> jima: I'd put that as a low priority but if you can't get it done tonight, I'd wait until after the release. 16:42 < jima> 'k. 16:42 < nim-nim> ticket 215 anyone? 16:42 < mmcgrath> .ticket 215 16:42 < mmcgrathbot> mmcgrath: #215 (New mailing list request: fedora-fonts-bugs-list) - Fedora Infrastructure - Trac 16:42 < mmcgrath> nim-nim: that one's on me. I'll put that request in now. 16:43 < nim-nim> mmcgrath: thanks! 16:43 * mmcgrath has been a little busy :) 16:43 * nim-nim understands 16:43 < mmcgrath> I'd also like to get the rest of the hosted repos created that are in the queue by tonight. 16:43 < nim-nim> still, nim-nim needs to get his SIG in shape too 16:44 < nim-nim> many thanks! 16:44 -!- JSchmitt [n=s4504kr@fedora/JSchmitt] has quit "Konversation terminated!" 16:44 < mmcgrath> Ok, anyone have anything else? 16:45 < mmcgrath> If not I'll close the meeting in 30. 16:45 < mmcgrath> 15 16:45 < mmcgrath> 5 16:45 < mmcgrath> Good luck to us all next week :) 16:46 -!- mmcgrath changed the topic of #fedora-meeting to: Infrasturcture -- Meeting Closed. 16:46 -!- fchiulli [i=824c4010@gateway/web/cgi-irc/ircatwork.com/x-a6b81cd860eb6651] has left #fedora-meeting [] 16:46 < mmcgrath> Thanks for coming everyone.
Attachment:
pgpwWMhBV5Uh2.pgp
Description: PGP signature
_______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
