On Tue, 10 Mar 2015, Blinick, Stephen L wrote: > Hi -- was running some baselines on 0.93 and wanted to check, will > wip-auth changes go into Hamer or in a later release? (Sorry if I > missed the discussion in one of the perf meetings last month) It will go in after hammer. We can backport as needed. sage > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson > Sent: Wednesday, February 4, 2015 4:25 PM > To: Sage Weil; Andreas Bluemle > Cc: Blinick, Stephen L; ceph-devel@xxxxxxxxxxxxxxx > Subject: Re: wip-auth > > Hi All, > > I completed some tests with wip-auth to the memstore on ubuntu earlier today. Basic gist of it is that the improvements in wip-auth help but don't quite get us to what can be achieved with auth disabled. RHEL7 (without auth) continues to do very well in latency bound situations. > Next up will be to see if how much this matters when testing against on SSDs. > > Here are the results: > > sync 4k object writes > ===================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.99 1007 > Wip-auth Ubuntu 14.04 Yes 0.81 1237 > master Ubuntu 14.04 No 0.64 1549 > Wip-auth Ubuntu 14.04 No 0.65 1563 > master RHEL7 No 0.32 3158 > > sync 4k object reads > ==================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 0.59 1695 > Wip-auth Ubuntu 14.04 Yes 0.41 2409 > master Ubuntu 14.04 No 0.29 3425 > Wip-auth Ubuntu 14.04 No 0.29 3474 > master RHEL7 No 0.17 5853 > > 256 concurrent 4k object writes > =============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 40.39 6339 > Wip-auth Ubuntu 14.04 Yes 26.22 9763 > master Ubuntu 14.04 No 17.46 14662 > Wip-auth Ubuntu 14.04 No 17.34 14759 > master RHEL7 No 14.93 17139 > > 256 concurrent 4k object reads > ============================== > > Ceph OS Auth Avg Lat (ms) Avg IOPS > ---------------------------------------------------------------- > master Ubuntu 14.04 Yes 31.47 8134 > Wip-auth Ubuntu 14.04 Yes 19.81 12922 > master Ubuntu 14.04 No 12.82 19968 > Wip-auth Ubuntu 14.04 No 12.75 20080 > master RHEL7 No 12.04 21257 > > Mark > > On 01/30/2015 03:08 PM, Sage Weil wrote: > > Hi Andreas, > > > > It looks like that was a stale sha1, but the newer one was also > > broken. I've retested and it's working for me now. See latest > > wip-auth, > > sha1 0c21a7875059bef80842756dfb003f47cc2d66a6. > > > > Thanks! > > sage > > > > On Fri, 30 Jan 2015, Andreas Bluemle wrote: > > > >> Hi Sage, > >> > >> I tried to integrate wip-auth into my 0.91 build environment. > >> > >> I had not been able to start the cluster successfully: ceph-mon > >> crashes with a segmentation fault in CryptoKey::encrypt() (see > >> attachment). > >> > >> This happens when linking with libnss or libcryptopp (version 5.6.2). > >> > >> I created the patch to add wip-auth based on github pull request 3523 > >> and was able to use this patch directly with 0.91 with only a minor > >> adaptation for common/ceph_context.h: > >> the 0.91 version of ceph_context.h did not know anything about the > >> experimental "class CephContextObs". > >> > >> wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933. > >> > >> As my build environment is rpm, I had to modify the invocation of the > >> configure script in the spec file instead of the do_autogen.sh > >> script. > >> > >> > >> Best Regards > >> > >> Andreas Bluemle > >> > >> > >> On Tue, 27 Jan 2015 09:18:45 -0800 (PST) Sage Weil <sweil@xxxxxxxxxx> > >> wrote: > >> > >>> I spent some time focusing on just CryptoKey::encrypt(). I > >>> benchmarked 100,000 encrypts of 128 bytes and got, at baseline, > >>> > >>> cryptopp: 100000 encoded in 0.655651 > >>> libnss : 100000 encoded in 1.288786 > >>> > >>> Ouch! With a (fixed) version of my earlier patch that avoids > >>> uselessly copying the input buffer: > >>> > >>> 100000 encoded in 1.231977 > >>> > >>> With a patch that puts the key structures in CryptoKey instead of > >>> recreating them each time: > >>> > >>> 100000 encoded in 0.396208 -- ~70% improvement over original > >>> > >>> This is pushed to wip-auth. There's also a patch that caches key > >>> structs for crypopp.. it now takes > >>> > >>> 100000 encoded in 0.440758 -- ~33% improvement over original > >>> > >>> (Not that almost anybody will ever care; we use libnss by default > >>> for both rpm and deb distros.) > >>> > >>> So, yay, nss is now a bit faster. What I'm not completely certain > >>> about is whether the structures I've preserved are truly stateless > >>> (and can be shared across threads, etc.). They encrypt/decrypt > >>> methods are const so, if the libraries are const-correct, it should > >>> be fine... but perhaps someone familiar with nss and/or crypto++ can > >>> review this? > >>> > >>> This is pushed to the latest wip-auth branch: > >>> > >>> https://github.com/ceph/ceph/commits/wip-auth > >>> > >>> Andreas and Stephen, what effect does this have on your numbers? > >>> > >>> Thanks! > >>> sage > >>> > >>> > >>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > >>> > >>>> Good to know, I was wondering why the spec file defaulted to > >>>> lib-nss.. the dpkg-build for debian packages just uses whatever > >>>> configuration you had built, and I believe that will use > >>>> libcryptopp if the dependency is installed on the build machine > >>>> (last I looked). > >>>> > >>>> I forgot to mention the numbers below were based on v.91. > >>>> > >>>> Thanks, > >>>> > >>>> Stephen > >>>> > >>>> -----Original Message----- > >>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Sage Weil > >>>> Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L > >>>> Cc: andreas.bluemle@xxxxxxxxxxx; ceph-devel@xxxxxxxxxxxxxxx > >>>> Subject: RE: wip-auth > >>>> > >>>> On Mon, 26 Jan 2015, Blinick, Stephen L wrote: > >>>>> I noticed that the spec file for building RPM's defaults to > >>>>> building with libnss, instead of libcrypto++. Since the > >>>>> measurements I'd done so far were from those RPM's I rebuilt with > >>>>> libcrypto++.. so FWIW here is the difference between those two on > >>>>> my system, memstore backend with a single OSD, and single client. > >>>>> > >>>>> Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7 > >>>>> Kernel: 3.10.0-123.13.2.el7 > >>>>> > >>>>> 100% 4K Writes, 1xOSD w/ Rados Bench > >>>>> libnss | > >>>>> Cryptopp # QD IOPS Latency(ms) | > >>>>> IOPS Latency(ms) IOPS Improvement % 16 > >>>>> 14432.57 1.11 | 18896.60 0.85 > >>>>> 30.93% 100% 4K Reads, 1xOSD w/ Rados Bench libnss | Cryptopp # QD > >>>>> IOPS Latency(ms) | IOPS Latency(ms) IOPS Improvement % 16 > >>>>> 19532.53 0.82 | 25708.70 0.62 31.62% > >>>> > >>>> Yikes, 30%! I think this definitely worth some effort. We > >>>> switched to libnss because it has the weird government > >>>> certfiications that everyone wants and is more prevalent. crypto++ > >>>> is also not packaged for Red Hat distros at all (presumably for > >>>> that reason). > >>>> > >>>> I suspect that most of the overhead is in the encryption context > >>>> setup and can be avoided with a bit of effort.. > >>>> > >>>> sage > >>>> > >>>> > >>>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Stephen > >>>>> > >>>>> -----Original Message----- > >>>>> From: ceph-devel-owner@xxxxxxxxxxxxxxx > >>>>> [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Sage Weil > >>>>> Sent: Thursday, January 22, 2015 4:56 PM > >>>>> To: andreas.bluemle@xxxxxxxxxxx > >>>>> Cc: ceph-devel@xxxxxxxxxxxxxxx > >>>>> Subject: wip-auth > >>>>> > >>>>> Hi Andreas, > >>>>> > >>>>> I took a look at the wip-auth I mentioned in the security call > >>>>> last week... and the patch didn't work at all. Sorry if you > >>>>> wasted any time trying it. > >>>>> > >>>>> Anyway, I fixed it up so that it actually worked and made one > >>>>> other optimization. It would be great to hear what latencies you > >>>>> measure with the changes in place. > >>>>> > >>>>> Also, it might be worth trying --with-cryptopp (or --with-nss if > >>>>> you built cryptopp by default) to see if there is a difference. > >>>>> There is a ton of boilerplate setting up encryption contexts and > >>>>> key structures and so on that I suspect could be cached (perhaps > >>>>> stashed in the CryptoKey struct?) with a bit of effort. See > >>>>> > >>>>> > >>>>> https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L2 > >>>>> 13 > >>>>> > >>>>> sage > >>>>> -- > >>>>> To unsubscribe from this list: send the line "unsubscribe > >>>>> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx > >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>>> > >>>>> > >>>> -- > >>>> To unsubscribe from this list: send the line "unsubscribe > >>>> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>> -- To unsubscribe from this list: send the line "unsubscribe > >>>> ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx > >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> -- > >> Andreas Bluemle mailto:Andreas.Bluemle@xxxxxxxxxxx > >> ITXperts GmbH http://www.itxperts.de > >> Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 > >> D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > >> > >> Company details: http://www.itxperts.de/imprint.htm > >> > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
