============================================ #fedora-meeting: Infrastructure (2011-12-08) ============================================ Meeting started by nirik at 19:00:01 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2011-12-08/infrastructure.2011-12-08-19.00.log.html Meeting summary --------------- * Robot Roll Call (nirik, 19:00:01) * New folks introductions and Apprentice tasks (nirik, 19:03:08) * serverbeach / collab / hosted status (nirik, 19:06:25) * ibiblio machines status (nirik, 19:12:28) * gather more info/options for torrent seed/trackers. (nirik, 19:19:25) * Upcoming Tasks/Items (nirik, 19:28:19) * 2011-12-08 - Fedora 14 end of life. (nirik, 19:28:31) * 2011-12-12 - migrate to collab03/04 (evening) (tenative) (nirik, 19:28:31) * 2011-12-14 - migrate hosted (tenative) (nirik, 19:28:31) * 2011-12-22 - contact peer1 about retirement schedule for old machines. (nirik, 19:28:31) * 2011-12-23 to 2012-01-02 - rh shutdown week. (nirik, 19:28:31) * 2012-01-10 - drop inactive maintainers from pkgdb acls. (nirik, 19:28:31) * 2012-01-13 to 2012-01-15 - FUDCON blacksberg (nirik, 19:28:33) * 2012-02-31 - fas 0.8.11 final release. (nirik, 19:28:37) * 2012-02-14 to 2012-02-28 - F17 Alpha Freeze (nirik, 19:28:39) * Meeting tagged tickets (nirik, 19:29:47) * NFS outage (nirik, 19:30:21) * LINK: http://www.cbsnews.com/8301-201_162-57339465/2-shot-dead-at-va-tech-gunman-sought/ (skvidal, 19:47:34) * Open Floor (nirik, 19:47:56) Meeting ended at 20:38:53 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * skvidal (206) * nirik (177) * dgilmore (37) * bress (35) * abadger1999 (27) * jds2001 (24) * mdomsch (11) * smooge (5) * jnalley (5) * zodbot (4) * StylusEater (1) * jac1bat0 (1) * jsmith (1) * lmacken (0) * ricky (0) * Codeblock (0) -- 19:00:01 <nirik> #startmeeting Infrastructure (2011-12-08) 19:00:01 <zodbot> Meeting started Thu Dec 8 19:00:01 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <nirik> #topic Robot Roll Call 19:00:01 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:01 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge 19:00:10 * skvidal is here 19:01:03 <nirik> morning everyone. 19:01:10 * nirik will wait a few min for folks to wander in 19:01:26 * jnalley is here 19:01:26 <jac1bat0> hello nirik, everybody 19:01:32 * abadger1999 here 19:01:33 * jsmith is lurking 19:02:08 * mdomsch is lurking too 19:03:08 <nirik> #topic New folks introductions and Apprentice tasks 19:03:13 <nirik> ok, lets dive in. 19:03:25 <nirik> any new folks or apprentices care to discuss anything? 19:03:33 <nirik> or introduce themselves? 19:03:59 <nirik> I'm going to try and add some more easyfix tickets soon... if I can get fires quenched. 19:04:27 <jnalley> hi, i introduced myself on the list a while back but haven't really started working on a ticket since then. just today i contacted some ticket owners about helping out on otherwise stale tickets that hopefully will be easy for an fi-n00b. 19:04:43 <nirik> jnalley: cool. Welcome. 19:04:50 <jnalley> and it does't help that i also changed my nick from the less-intelligble nick ke4zvu3 19:04:54 <nirik> do hang out in #fedora-admin and/or #fedora-noc and chime in as you can. 19:05:02 <jnalley> i have, do, and will 19:05:05 <nirik> lost your ham license? ;) 19:05:18 <jnalley> haha, no, just changed it to be uniform with my FAS name 19:05:28 <nirik> oh good. 19:06:01 <nirik> any other introductions or issues with apprentice tasks? 19:06:17 * nirik moves on. 19:06:25 <nirik> #topic serverbeach / collab / hosted status 19:06:40 <nirik> so, we have our new new hardware in, we have installed on it. 19:06:51 <nirik> I've sent out an announcement about moving collab on next monday. 19:06:59 <nirik> from collab01 to collab03. 19:07:19 <nirik> Hopefully that will be a pretty easy migration. 19:07:42 <nirik> Also, I am going to send out an announcement later today about moving hosted on next wed... from hosted01 to hosted03. 19:08:10 <nirik> If anyone has ideas for stress testing the new machines before we migrate to them, please do speak up. 19:08:47 <nirik> right now, I am thinking of just moving hosted pretty much as is, because I want to get it off rhel5/old machine. 19:09:04 <skvidal> nirik: maybe just rsync back and forth from the systems 19:09:09 <skvidal> to fill up the disks - to test the range 19:09:19 <nirik> then we can schedule a lists.fedorahosted.org move later along with a more targeted move of projects to a new setup with cnames. 19:09:35 <nirik> skvidal: yeah, I have been running rsyncs... they do have the data from the live/prod boxes on them. 19:10:06 <nirik> I need to finish the migration script for hosted and then we can start hitting it for testing again. 19:10:18 <nirik> (via a /etc/hosts change) 19:11:17 <nirik> also, it's worth noting that I setup hosted03/04 and collab03/04 with drbd... It means if the primary dies, we can bring up the data on the secondary pretty easily and not have much loss. 19:12:00 <nirik> so, any questions or concerns on all that? or shall we move on? 19:12:28 <nirik> #topic ibiblio machines status 19:12:40 <nirik> we have a new ibiblio03 coming in next week or so I think. 19:12:55 <nirik> we have also started moving things from ibiblio01->02. 19:13:04 <nirik> (which ran into some anoying ipv6 pain yesterday. ;( 19:13:38 <nirik> we have 3 things still on 01... 19:13:49 <nirik> smtp-mm03 (which can be migrated in the next few days) 19:14:01 <nirik> backup02 (which I want to migrate to ibiblio03 once it's in) 19:14:05 <nirik> and torrent01. 19:14:16 <skvidal> :( to that last one 19:14:16 <nirik> what will it take to migrate torrents to torrent02? 19:14:24 <skvidal> nirik: a torrent client from this decade? 19:14:39 <nirik> yeah, that would be nice. 19:14:44 <skvidal> we're sorta up shit creek when it comes toa torrent seed and tracker that has the same features 19:14:46 <skvidal> I can get one working 19:14:48 <skvidal> but not the logging 19:14:51 <skvidal> and stats tracking 19:14:58 <skvidal> that btseed/ bttrack currently offers 19:15:06 <nirik> fun. Can we just rebuild the thing we have now for now? 19:15:22 <skvidal> I can probably force it to work on el6 19:15:24 <nirik> and look at moving to magnet or something. 19:15:32 <skvidal> magnet links will provide even less info 19:15:44 <skvidal> it lets us just distribute out the link info - which is nice 19:15:52 <skvidal> but won't help us provide stats on if they are being used :( 19:16:05 <nirik> ok. wasn't herlo investigating some of the options? 19:16:25 <skvidal> nirik: istr, yes - but last I heard the options were all full of suck.. 19:16:34 <skvidal> I'll ask him if he had a final report or not 19:16:41 <nirik> worth asking on the devel/infra lists for ideas? 19:16:59 <skvidal> <shrug> I suspect we 19:17:10 <skvidal> 'll get a dozen comments of an unhelpful nature 19:17:18 <skvidal> that's not cynicism - just reality 19:17:20 <nirik> well, I'd like to move things off ibiblio01 so we can rhel6ize it, but we can see. 19:17:28 <nirik> yeah, probibly the case. 19:17:35 <skvidal> it seems like outside of the tpb no one really USES torrents much 19:17:55 <nirik> perhaps we could contact them and get them to open source their stuff? ;) 19:18:00 <skvidal> hahahaha 19:18:03 <skvidal> you're funny 19:18:28 <skvidal> oh 19:18:28 <skvidal> right 19:18:33 <skvidal> I knew I had forgotten something 19:18:43 <skvidal> the other issue we were hoping to work around 19:18:45 <skvidal> torrent + ipv6 19:18:55 <skvidal> which the existing client does not support, at all. 19:19:09 <nirik> yeah. ;( 19:19:25 <nirik> #info gather more info/options for torrent seed/trackers. 19:19:34 <skvidal> if we could dump the data gathering requirement 19:19:40 <skvidal> we could make something else work, probably quickly, too 19:19:56 <nirik> So what do we use that data for? 19:20:01 <nirik> I guess spins popularity? 19:20:56 <skvidal> and in general 19:21:02 <skvidal> just 'how much stuff got downloaded' 19:21:08 <nirik> well, the statistics page doesn't really use them anymore... 19:21:13 <skvidal> nod 19:21:23 <mdomsch> skvidal: opentracker still not up to snuff? 19:21:25 <mdomsch> I haven't looked in a while 19:21:27 <nirik> but the spins.fp.o uses them a lot 19:21:39 <mdomsch> last I knew, it still had the split v4 and v6 daemons 19:21:41 <skvidal> mdomsch: stats crash iirc 19:21:48 <mdomsch> skvidal: I think that's fixed now 19:21:52 <mdomsch> per bugzilla 19:22:08 <mdomsch> worth a look at least 19:22:23 <skvidal> mdomsch: I think this is what herlo was looking into 19:23:04 <nirik> I guess if it comes down to it, we can just move torrent01 over to ibiblio02 as a rhel5 if we can't come up with any better options. 19:23:44 <nirik> ok, gather more info, try and find a non sucky option... 19:23:46 <skvidal> a question I'd love an answer to 19:23:53 <skvidal> do we really NEED the torrent anymore? 19:24:04 <skvidal> can we just deprecate it? would the sky fall down on us? 19:24:08 <nirik> well, spins are currently not widely mirrored 19:24:13 <nirik> they are torrent only. 19:24:22 <skvidal> torrent _only_? 19:24:23 <skvidal> no 19:24:24 <nirik> (well, they are in the alt mirror component, but very few mirrors mirror that) 19:24:26 <skvidal> I don't think that's true 19:24:28 <skvidal> okay 19:24:31 <skvidal> so not torrent _only_ 19:25:12 <nirik> yeah, but a pretty small number of mirrors. 19:25:41 * nirik would be happy to punt that decision to the Board or fesco. ;) 19:25:43 <skvidal> how large is the consume population? 19:26:26 <nirik> from looking at the torrent stats, not all that vast. 19:26:40 <mdomsch> opentracker is in the trees, and is lightly maintained (one spec patch since Jan) 19:27:27 <nirik> lets try and gather more info, and revisit next week... 19:28:03 <dgilmore> hola 19:28:11 <nirik> hey dgilmore 19:28:19 <nirik> #topic Upcoming Tasks/Items 19:28:27 <nirik> flood ahead: 19:28:31 <nirik> #info 2011-12-08 - Fedora 14 end of life. 19:28:31 <nirik> #info 2011-12-12 - migrate to collab03/04 (evening) (tenative) 19:28:31 <nirik> #info 2011-12-14 - migrate hosted (tenative) 19:28:31 <nirik> #info 2011-12-22 - contact peer1 about retirement schedule for old machines. 19:28:31 <nirik> #info 2011-12-23 to 2012-01-02 - rh shutdown week. 19:28:31 <nirik> #info 2012-01-10 - drop inactive maintainers from pkgdb acls. 19:28:33 <nirik> #info 2012-01-13 to 2012-01-15 - FUDCON blacksberg 19:28:37 <nirik> #info 2012-02-31 - fas 0.8.11 final release. 19:28:39 <nirik> #info 2012-02-14 to 2012-02-28 - F17 Alpha Freeze 19:28:48 <nirik> anything anyone would like to discuss upcoming? 19:29:43 <nirik> ok, moving along then. 19:29:47 <nirik> #topic Meeting tagged tickets 19:30:10 <nirik> none. 19:30:21 <nirik> #topic NFS outage 19:30:25 <dgilmore> :( 19:30:28 <nirik> We are currently in a nfs outage. ;( 19:30:41 <nirik> it looks like it was caused by a network hiccup to the backend storage. 19:31:05 <nirik> Not sure what we can do to avoid this kind of thing moving forward, but it's an anoying SPOF 19:31:26 <dgilmore> yeah 19:31:38 <dgilmore> iscsi ftw 19:32:22 <nirik> so, if we have ideas for clustered filesystems or the like, we can investigate. I'm not sure I trust any of them with our data yet tho. 19:32:33 <dgilmore> yeah 19:32:38 <skvidal> all of the clustered fs will be an issue 19:32:43 <skvidal> b/c of ownerships afaict 19:32:47 <nirik> yeah 19:32:49 <skvidal> maybe gfs2 19:32:55 <dgilmore> skvidal: yeah 19:33:03 <skvidal> but I'm not sure that's going to not require an arseload of infrastructure that we do not have 19:33:09 * nirik hasn't looked at gfs2 lately, but long ago I remember it being a royal pain to setup. 19:33:27 <dgilmore> maybe a clusered fs would help with a network glitch 19:33:38 <mdomsch> gluster ? :-) 19:33:44 <dgilmore> but its a whole different ball of fish 19:33:59 * nirik notes that RH announced a storage product today... 19:34:10 <skvidal> gluster 19:34:11 <mdomsch> nirik: yup 19:34:14 <skvidal> gluster has the perms issues 19:34:32 <skvidal> it can do normal posix 19:34:32 <nirik> no acls, right? 19:34:35 <skvidal> but not extended acls 19:34:45 <skvidal> and no selinux - though that may have changed very recently 19:34:50 <mdomsch> ask ric if that's coming... 19:34:52 <dgilmore> we dont do acls 19:34:54 <nirik> not sure we use acls on /mnt/koji 19:35:00 <dgilmore> nirik: we dont 19:35:14 <skvidal> mdomsch: I've asked the gluster folks and it is tricky 19:36:14 <nirik> so, that might be something to look into... would need to setup a testbed and make sure it doesn't blow up and that we understand it. 19:36:51 <nirik> not sure what we could do short term. ;( 19:37:12 <dgilmore> yeah 19:37:20 <dgilmore> not sure it would really help 19:37:32 <dgilmore> gluster is userland right 19:37:40 <dgilmore> on top of something else? 19:37:52 <skvidal> dgilmore: it's userland - yes - but it replicates across the network 19:38:17 <skvidal> dgilmore: you then mount it from userland onto your clients - I want to say it is fuse 19:38:20 <skvidal> on the client end 19:38:22 <nirik> yes, fuse 19:38:27 <dgilmore> skvidal: right 19:38:36 <dgilmore> skvidal: which would do nothing in this case 19:38:44 <skvidal> ?? 19:38:46 <skvidal> it would do nothing? 19:38:53 <dgilmore> since the issue was that the iscsi storage blipped 19:38:57 <skvidal> if we had 2 units setup as mirrors 19:38:59 <dgilmore> and ext4 got messed up 19:39:03 <skvidal> and each of them had independent backends? 19:39:17 <dgilmore> skvidal: so if we got another blip likely get the same 19:40:03 <skvidal> oh - if they are both backended on the same storage network 19:40:06 <skvidal> yah - I see 19:40:18 <dgilmore> skvidal: yeah. 19:40:29 <skvidal> so do we have a place to replicate the nfs data to? 19:40:41 <dgilmore> skvidal: bnfs01 19:40:49 <dgilmore> but it takes me a day to rsync to it 19:40:56 <nirik> drbd! 19:40:57 <skvidal> would drbd 19:40:58 * nirik runs 19:41:00 <skvidal> exactly 19:41:03 <skvidal> will it scale to that size? 19:41:09 <dgilmore> 12T 19:41:20 <dgilmore> no idea 19:41:27 <nirik> it can, as long as the net can keep up to the disk i/o 19:41:49 <nirik> which usually isn't a problem 19:42:00 <skvidal> are bnfs01 and koji's nfs backended to different devices? 19:42:08 <skvidal> or is it all onto the netapp? 19:42:08 <dgilmore> yes 19:42:19 <skvidal> yes to which one 19:42:23 <nirik> different devices 19:42:27 <dgilmore> bnfs01 is 1tb drives attached directly using disk shelves 19:42:37 <dgilmore> nfs01 is equalogic over iscsi 19:42:50 <skvidal> and how close to end of life is bnfs01, it's disks, etc? 19:43:03 <dgilmore> both are ~1 year old 19:43:12 <nirik> 2013-01-17 19:43:16 <skvidal> okay, great 19:43:20 <skvidal> wait 19:43:24 <skvidal> Jan 2013? 19:43:29 <skvidal> so a bit over a year? 19:43:33 <skvidal> like 13 months? 19:43:34 <nirik> yep 19:43:39 <skvidal> bleah 19:43:42 <dgilmore> nirik: that doesnt seem right 19:43:43 <nirik> but we can extend. 19:43:54 <nirik> unless it's labeled wrong. it could be. 19:44:07 <jds2001> arent they normally 3 years? 19:44:08 * nirik can check for sure. 19:44:14 <nirik> jds2001: yep 19:44:18 <dgilmore> i thought normally 5 years 19:44:21 * jds2001 checks into the cheap seats :) 19:44:24 <skvidal> jds2001: well if ~1yr old == 2yrs 19:44:24 <nirik> 3 is normal 19:44:30 <skvidal> jds2001: then.. :) 19:44:43 <jds2001> heh 19:44:49 <nirik> in any case... one pain of moving to drbd is that we need somewhere to shuffle the data, make the drbd and then shuffle back. ;) 19:44:49 <dgilmore> im often wrong 19:44:57 <skvidal> nirik: nod 19:45:13 <skvidal> nirik: we could probably beg for tmpspace on the netapp 19:45:16 <skvidal> nirik: for a few weeks 19:45:17 <nirik> yeah. 19:45:39 * nirik has used heartbeat+drbd for nfs servers in the past. 19:45:52 <skvidal> you'd have to remount everything 19:46:00 <skvidal> b/c the filehandles would all change 19:46:05 <nirik> yeah. 19:46:07 <skvidal> it's doable - but with our static mounts its harder 19:46:09 * skvidal likes autofs 19:46:20 <skvidal> but I'm probably one of the 4 people in the universe who does 19:46:29 <jds2001> skvidal: that makes one of us :) 19:46:43 <nirik> anyhow, thats also longer term... perhaps we could have a nice 'nuke SPOFs' talk at fudcon too. 19:46:54 <skvidal> nirik: we had that talk at the last fudcon :) 19:47:11 <nirik> yeah, and we can keep doing it until we get it right! :) 19:47:19 <skvidal> hah 19:47:22 <skvidal> oh and remember everyone 19:47:28 <skvidal> bring your body armor to fudcon 19:47:34 <skvidal> http://www.cbsnews.com/8301-201_162-57339465/2-shot-dead-at-va-tech-gunman-sought/ 19:47:40 * jds2001 prepares for the 'nuke SPOF' talk at fudcon 2060 :D 19:47:41 <nirik> yeah, crazy. 19:47:49 <nirik> anyhow... 19:47:49 <dgilmore> skvidal: im all for eliminating spof 19:47:56 <nirik> #topic Open Floor 19:48:05 <nirik> anyone have anything for open floor? 19:48:48 * jds2001 hasn't been around in infra a ton lately 19:48:53 * jds2001 apologizes :/ 19:48:55 <nirik> jds2001: welcome back. ;) 19:49:05 <skvidal> there's a lot of tasks 19:49:07 <skvidal> which need doing 19:49:11 <skvidal> which need time 19:49:16 * nirik gets the list of things we assigned to jds2001 while he wasn't around. ;) 19:49:18 <skvidal> if anyone wants to work on projects 19:49:21 <skvidal> there are a bunch of them 19:49:50 <jds2001> well all of this talk of collab and mailing lists is near and dear to my heart. 19:50:20 <jds2001> though i really dont want to work on it for 14-18 hours straight like last time :) 19:50:28 <nirik> we still do need to get the last lists at rh moved too someday. 19:50:37 <skvidal> nirik: like what? 19:50:45 <jds2001> epel and such 19:50:46 <nirik> epel-devel, some of the ambassadors lists. 19:51:26 <nirik> I am hoping the collab move will be very easy. I have fewer hopes for the fedorahosted one... 19:52:00 <jds2001> well, the hosted move is similar to moving from rh, right? 19:52:10 <abadger1999> nirik: Do you want to update on what bressers said? 19:52:13 <jds2001> that wasn't terribly hard, except a few glitches. 19:52:44 <nirik> jds2001: the moving lists.fedorahosted -> collab? yeah, that will be just like that, but with another domain tossed into the mix. 19:52:46 <abadger1999> nirik: I was wondering specifically if he'd said anything about how much diversity wold be good/bad. 19:52:54 <nirik> abadger1999: oh yeah, sorry, I meant to mention that. 19:53:14 <jds2001> nirik: and having control of both sides makes it even easier, really. 19:53:16 * abadger1999 is sure that 2 chars is good, but unsure above that. 19:53:40 <nirik> so, he said: 19:53:45 <nirik> "I would say in the short term, enact some of the above rules. They won't 19:53:46 <nirik> hurt and will certainly prevent really bad passwords. They won't solve the 19:53:46 <nirik> real problem though." 19:54:04 <nirik> sadly, he is not going to be at fudcon. ;( 19:54:17 <nirik> perhaps we could invite him for next weeks meeting and discuss this a bit more? 19:54:41 <abadger1999> Sure. 19:55:13 * nirik sees if he's around now. guess not. 19:55:22 <skvidal> abadger1999: what is 'the real problem'? 19:55:41 <nirik> skvidal: having passwords instead of 2factor or the like. 19:55:45 <skvidal> ah 19:55:57 <skvidal> I do have an update on the 2fa stuff 19:56:22 <skvidal> I worked on the pam_linotp module - I got it building and it submits the 2fa input over the wire to a cgi which can verify it 19:56:46 <nirik> cool. 19:56:53 <skvidal> so that works just fine. to make 2fa happen on EVERY ssh login, though, we'll need to modify sshd 19:57:00 <skvidal> with patches from dwmw2 19:57:18 <skvidal> if we just want them on sudo, though, that should be fine right now 19:57:28 <nirik> so what can be used as the other factor there? googleauth? yubikey? 19:57:51 <jds2001> yubikey is simple and i already have one :) 19:57:53 <skvidal> right now it would be totp 19:58:14 <skvidal> but, ostensibly, you could verify whatever in the cgi 19:58:17 * jds2001 hasnt even looked at googleauth, but it seems cool. 19:58:35 <skvidal> since the string sizes are so massively different between totp and yubikeys 19:58:35 <skvidal> it should be possible to accept either 19:58:48 * dgilmore doesnt trust anything with google in the name 19:58:57 <nirik> dgilmore: it's open source. ;) 19:58:59 * skvidal shakes his head 19:59:13 <nirik> I'd be find with yubikeys for sudo, but want a pin + yubikey 19:59:20 <jds2001> dgilmore: at least its not cubsauth :D 19:59:20 <dgilmore> nirik: i know, I have an irrational distrust of anything google does 19:59:23 <dgilmore> open or not 19:59:25 <nirik> and it sounds like this linotp might be able to let us do whichever 19:59:33 <skvidal> nirik: all linotp offers is this 19:59:43 <skvidal> 1. a pam_module to submit the info to a cgi elsewhere 19:59:48 <abadger1999> I think having ssh key for login and 2fa auth for sudo would offer a lot over what we currently have. 19:59:52 <dgilmore> jds2001: id be ok with that 20:00:11 <nirik> abadger1999: +1. I'd like that 20:00:12 <skvidal> 2. some simple routines to verify various types of 2fa 20:00:28 <skvidal> to make something work for both totp and yubikey 20:00:35 <skvidal> we'd have to maintain it ourselves 20:00:43 <skvidal> though I suspect 20:00:45 <skvidal> if we do this 20:00:51 <skvidal> other people would be interested 20:00:57 <nirik> yeah. 20:01:21 <dgilmore> skvidal: i agree 20:01:41 <skvidal> modifying the pam module to get it working took some minor patching - nothing serious 20:01:49 <skvidal> and there are a few things in the module which make me grumpy 20:01:51 <nirik> is it packaged up at all? 20:01:57 <skvidal> oh god no 20:02:04 <nirik> ok. 20:02:06 <skvidal> and to be fair 20:02:15 <skvidal> packaging up all of lin_otp seems like a bad idea 20:02:23 <skvidal> they want to be a one-stop shop for all sorts of auth stuff 20:02:25 <nirik> ok, we just need the pam module? 20:02:29 <skvidal> and integrating it into fas would be a mess 20:02:47 <skvidal> pammodule + whatever cgi the pam module submits info to 20:03:06 <skvidal> they have a cgi which is django based and involves all their other admin-interface 20:03:07 <nirik> yeah. 20:03:15 <skvidal> which would probably make us want to kill ourselves to keep it going 20:03:25 <nirik> right, so perhaps we want to look at forking and cleaning up/simplifying this. 20:03:30 <skvidal> right 20:03:55 <skvidal> so let me confirm a few things 20:03:57 <nirik> abadger1999: you had some questions for bress ? 20:03:57 <skvidal> b/c that will help 20:04:02 <jds2001> has anyone had disucssions with upstream about that? 20:04:14 <jds2001> i.e. making a "lin_otp lite" if you will? 20:04:18 <skvidal> jds2001: they are using the split-source/commerical modules thing 20:04:25 <jds2001> :/ 20:04:35 <skvidal> jds2001: so... no 20:04:56 <skvidal> oh and one more hitch to using a lot of their code 20:05:03 <skvidal> some of it is agpl 20:05:05 <abadger1999> bress: hey, so about checking password strength by eliminating passwords of only a few different characters.... is there some number of different characters that makes things worse instead of better? 20:05:06 <nirik> ugh 20:05:32 * nirik notes we are over time, but I don't think there is a meeting after us if we want to keep going. 20:05:46 <skvidal> I'd like to get confirmation on a few things 20:05:53 <skvidal> nirik: after abadger1999 and bress are finishe 20:05:54 <skvidal> d 20:05:57 <nirik> skvidal: ok 20:07:03 <abadger1999> bress: My back of the envelope calculation is ensuring two separate characters for standard passwords has little drawback -- but my math skills aren't up to tackling how larger diversity requirements affect the ability to brute force passwords 20:07:03 <bress> abadger1999: I mailed Kevin Fenzi about this a while back. In general, password policies make things worse. The real answer is some sort of 2 factor system. In general though, you're not talking about reducing your corpus by all that much. 20:07:16 <abadger1999> k. 20:07:20 * nirik is kevin, FYI. ;) 20:07:24 * jds2001 thinks that no matter what technical restrictions we put on passwords, we'll always have someone choose something like AbC123$ 20:07:31 <abadger1999> <nod> 20:07:39 <skvidal> jds2001: shit, now I have to go change my password! 20:07:46 <nirik> the more restrictions also the worse documentation looks 20:07:50 <bress> I wouldn't be as worried about someone using a shitty password as I would be making sure an attacker can't brute force any of our systems. If you allow more than say 6 logins per minute, we're doing it wrong. 20:08:09 <bress> skvidal: Just go with AbC1234$ 20:08:14 <nirik> your password must be at least 12 characters, unless it's monday in which case you need to use 2 non repeating lower case letters, unless it's a full moon, in which case... 20:08:22 <StylusEater> bress: +1 on the login comment ... ;-) 20:08:30 <abadger1999> bress: So ensuring diversity of characters is going to hit, "what will our users stand for" quicker than "you've reduced your possible combinations too mch". 20:08:40 <bress> abadger1999: Yeah 20:08:48 <abadger1999> bress: thanks. 20:08:50 <bress> abadger1999: It could be wise to point folks at diceware. 20:09:07 <bress> Since picking a new password is really hard for people. 20:09:31 <nirik> yeah, we did link to that in our change announcement. 20:10:03 <skvidal> abadger1999: how hard would it be to modify our fas web logins so we require password + 2fa? 20:10:07 <bress> But I'd say the real answer here is going to be some sort of 2 factor auth. 20:10:29 <abadger1999> bress: That's nice -- smooge had also suggested ading javascript to our password reset that does something similar. 20:10:42 <abadger1999> As a password suggestion for people. 20:10:45 <smooge> working on that 20:10:46 <abadger1999> <nod> 20:10:48 <bress> Yeah, that's a good idea. 20:11:04 <smooge> sorry for missing the meeting guys 20:11:13 <skvidal> bress: which places do you think 2fa should be used: on any ssh login? git commits? webapp logins? sudo? 20:11:26 <nirik> smooge: lucky for you, it's still going on. ;) 20:11:42 <abadger1999> skvidal: "allow" pretty easy -- it's a variant on the yubikey acceptance that's already in. 20:11:54 <abadger1999> skvidal: "require" is harder. 20:11:56 <bress> skvidal: That one is tricky. I'd probably say anywhere a password is needed. Things where an SSH key will do is probably OK. 20:12:05 <abadger1999> skvidal: "require for some people" even more. 20:12:08 <bress> skvidal: If people can't commit without jumping through hoops, they just won't commit. 20:12:39 <skvidal> bress: oh cmon - marriage rates alone disprove that <zing> 20:12:46 <bress> The trick isn't to ellimate risk, but to understand and control it. 20:13:05 <skvidal> so another question wrt to commits 20:13:10 * bress wonders how long skvidal has been waiting to use that one ;) 20:13:12 <smooge> so question. To get an idea of how bad our users practices are.. can we "anonymize" the hashes from before the new hash became standard and run jtr with rockyou top 10000 20:13:15 <abadger1999> skvidal: but none of it is something we lack the ability to do (even considering how time constrained we are) 20:13:43 <skvidal> would it help us to require people to connect with an ssh master connection - requiring 2fa for that connection duration 20:13:53 <smooge> then if we find a large number before and after we say "simple test" if you chose a password here you need to chose something else. 20:13:59 <skvidal> so they could connect once, stay on, and then not have to auth again until forced out 20:14:12 * skvidal is just brainstorming a bit 20:14:37 <nirik> I suspect we will get a lot of pushback on making pkg commits 2factor. 20:14:46 <bress> skvidal: That's an interesting concept. The only problem I can see is people forgetting to logout (or just never logging out to save time) 20:14:46 <skvidal> nod 20:14:55 <skvidal> bress: forgetting to logout we can actually handle 20:15:09 <skvidal> bress: in much the same way rh handles it when I neglect to disconnect from the vpn 20:15:10 <bress> nirik: Ahh, but we can make them 2 factor sneakily. Do ssh for auth, and have users sign their commits :) 20:15:18 <jds2001> the issue that i see is like bress said, we're making things harder. 20:15:20 <bress> skvidal: You disconnect from the VPN? :) 20:15:27 <skvidal> bress: it makes me 20:15:35 <skvidal> bress: every 10hours or so 20:15:39 <bress> skvidal: I can keep openvpn up for days. 20:15:44 <bress> skvidal: But that's another conversation. 20:15:47 <skvidal> yes 20:15:48 <skvidal> anyway 20:15:50 <skvidal> my point is 20:16:00 <skvidal> we could force all communication through a host (or a set of hosts) like fedorapeople 20:16:01 <nirik> yeah, signing commits would be nice/cool. 20:16:18 <skvidal> and make everyone go through a 2fa there first 20:16:53 <skvidal> we could kill connections > N hours/days old 20:17:24 <skvidal> nirik: as far as signing commits - are we expecting gpg signs? and what about systems not using git? ie: fedorahosted? 20:17:42 <abadger1999> git and bzr both support gpg sig 20:17:53 * abadger1999 doesn't know about hg ... svn doesn't 20:18:00 <nirik> I was mostly waiting on the fallout from the kernel.org git commit signing stuff... which as far as I know hasn't really reached a conclusion. 20:18:04 <bress> We'd be crazy not to sign commits. It gives so many levels of safely it's not even funny. 20:18:06 <nirik> (but it might have and I missed it) 20:18:57 <skvidal> well we're going to need gpg keys from everyone and we have A LOT more people and a lot less tightly knit people than the kernel devs 20:19:06 <nirik> I think we can get short term win by inflicting 2f on smaller groups (sysadmin-main, then possibly sysadmin*), then learn from that before trying to deal with any larger groups. 20:19:19 <skvidal> okay 20:19:38 <nirik> if we end up using gpg more, we could require it for some groups? 20:20:03 * jds2001 was required to use GPG to sign the CLA back in the day 20:20:10 <skvidal> so I grok how gpg helps for commits 20:20:17 <skvidal> but it does nothing for system security afaict 20:20:21 <jds2001> one of hte main reasons for changing that was the technical chanllenges of doing that. 20:20:24 <bress> We don't require gpg keys from everyone? 20:20:30 * bress had no idea 20:20:39 <nirik> bress: fas has a field for it, but there's no requirement for it as far as I know. 20:20:45 <abadger1999> jds2001: Well... you had to sign the cla to edit the wiki so.... 20:21:05 <abadger1999> yeah -- also, they're gpg fingerprints, not keys 20:21:13 <nirik> yeah, right. 20:22:08 <bress> Anyway, this conversation has gotten derailed. I guess to sum it up, a simple password policy is best, and should help get us to a point where two factor auth can be used. 20:22:42 <nirik> my thoughts: short term try and get linotp or something flexable based of it going... then try and modify fas to handle yubikey+pin, and/or googleauth and/or whatever, then use that for sudo at least, and move from there. 20:23:03 <bress> abadger1999: Also, there's nothing wrong with adding rules to the code and not really telling people. I don't think anyone is going to complain by saying "When I tried all A's it didn't work", they're look like an idiot :) 20:23:27 <abadger1999> bress: Now that's something I kinda like ;-) 20:23:55 * nirik notes if we had yubikey + pin, he would be ok with using that for sudo now, but we don't. 20:24:14 <smooge> as long as it allows hellokittyhellodoggiewhysoserious 20:24:38 <bress> smooge: You probably need to add a 4 to the end to make it secure. 20:24:59 <skvidal> nirik: okay 20:25:08 <skvidal> so the confirmations I needed are mostly answered 20:25:14 <skvidal> but I need a couple more details 20:25:17 <abadger1999> bress: Somewhat side note -- is it okay for us to send you best practice security questions? We do get requests to implement things that I have no idea how to evaluate the impact from time to time. 20:25:19 <nirik> fire away. 20:25:33 <bress> abadger1999: Certainly. I'm happy to help any way I can. 20:25:41 <skvidal> abadger1999: do we have a place we could stuff OTP "secrets" 20:25:50 <skvidal> abadger1999: in fas, or something related, I mean. 20:26:05 <abadger1999> bress: Excellent. I'll make use of your expertise I'm sure :-) 20:26:24 * nirik is sad bress isn't going to make it to fudcon. ;( 20:26:39 <bress> abadger1999: Heh, sure, that's what I call it too :) But if I don't know, I probably know someone who does. 20:26:42 <abadger1999> skvidal: We can push the shared secret into fas's config table like the yubikey stuff has. 20:26:46 <skvidal> bress: wrt to totp 2fa - do you know if it is even theoretically possible, given enough valid entries to take time + valid OTP and get the secret out of it? 20:28:15 <bress> skvidal: I can't answer that really. I've not seen much research done in that area. I would presume it's *possible* but probably not very practical. 20:28:20 <nirik> there was also a lwn article this week on google authenticator. There was mention there of a command line tool too. (although it required converstion) 20:28:23 <skvidal> bress: second question - if we wanted to get a pam module audited for sanity, etc - can we send it to you 20:28:30 <skvidal> nirik: the command line tool is trivial 20:28:32 <skvidal> nirik: so here's the thing 20:28:42 <skvidal> totp is really easy to implement 20:28:54 <skvidal> the only tricky bit that google has implemented is their secret bailout codes 20:28:57 <bress> skvidal: Sure, it would give me a good opportunity to work out some of the bugs in my code auditing guide. 20:29:19 <skvidal> and afaict the only thing they've done is just keep the list of bailout codes and check against them - there's no math - they just look them up 20:29:21 <bress> skvidal: Google has made a google authenticator pam module public. 20:29:29 <skvidal> bress: yes - and we can't use it 20:29:35 <skvidal> bress: it only auth's locally 20:29:39 <nirik> well, we don't want to use it. ;) 20:29:42 <bress> Yes. 20:29:47 <skvidal> which means we'd have to haqve secrets on EVERY machine 20:29:56 <skvidal> so what I found was this module called pam_linotp 20:30:08 <skvidal> and all it does is relay your otp onto a central cgi 20:30:18 <skvidal> which checks it and gives you a yes or no or 'error' 20:30:23 <skvidal> and that's it. 20:30:32 <bress> Ooohhh 20:30:32 <skvidal> it relays it using libcurl + ssl 20:30:44 <bress> That part sounds ugly 20:30:47 <skvidal> no kidding 20:30:48 <nirik> much better for our use case at least. 20:31:00 <skvidal> bress: but it';s not really worse than ldap auth when you think about it. 20:31:01 <nirik> (then we only need secrets on a few high security machines) 20:31:08 * skvidal points at nirik 20:31:09 <skvidal> exactly 20:31:18 <skvidal> then we have to bottle up the boxes with access to the secrets 20:31:34 <bress> skvidal: ldap doesn't run curl :) 20:31:42 <skvidal> bress: well this doesn't RUN it either 20:31:44 <skvidal> it uses libcurl 20:31:53 <bress> OK, that's a lot better. 20:31:55 <skvidal> but this is why Iwas asking about auditing the pam module 20:32:12 <skvidal> it has options to the module for ssl verify, etc 20:32:13 <bress> Yeah, giving it a good beating is probably in order. 20:32:33 * jds2001 has to run to a $DAYJOB meeting 20:32:36 <skvidal> I've done some curl programming before and setting up things like client side ssl certs for connecting to the cgi is pretty straightforward 20:33:37 <nirik> so we need a backend part that has a web interface right (or is in fas, but perhaps seperate would be better). so you could enroll new 2fa thing, setup pin, lost factor, etc... right? 20:33:44 <skvidal> nah 20:33:51 <skvidal> nirik: you'd do the web interface bit in fas 20:34:01 <skvidal> and this would just be able to talk to the fas db which has all the info 20:34:10 <skvidal> or hell, it could even just get a replicated copy every N minutes 20:34:14 <nirik> well, the advantage of non fas is that we could possible get other projects interested in helping/using it... 20:34:21 <skvidal> agreed 20:34:26 <skvidal> hence - replicated copy of the data it needs 20:34:29 <nirik> or... stupid idea: 20:34:32 <skvidal> but use fas for input 20:34:38 <skvidal> nirik: or big pile of files :) 20:35:06 <nirik> on the auth server, chain the other auth method? ie, have yubikey, googleauth local to that machine to run the check, and return to the linotp thing? 20:36:00 <skvidal> huh? 20:36:15 <nirik> yeah, nevermind. 20:36:21 <skvidal> you could make the pam_unix check required - for normal 'checking your password' thing - then have it prompt for the otp 20:36:32 <skvidal> just like a 2fa gmail login does now 20:36:42 <nirik> yeah. 20:36:46 <skvidal> which has the unfortunate result of allowing you to know if you have the right password 20:37:02 <nirik> anyhow, anything else we want to hash out now? or shall we call it a long meeting? 20:37:25 <skvidal> bress: what address for you? bress@rh? 20:37:32 <skvidal> nirik: yah - I think that's all I needed 20:37:40 <bress> skvidal: bressers@xxxxxxxxxx 20:38:08 <nirik> cool. Hopefully we can setup something nice and flexable. 20:38:15 <nirik> Any other open floor items? 20:38:50 <nirik> ok, thanks for coming everyone! 20:38:53 <nirik> #endmeeting
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
