Hi Alex,
So far we don't really know, but suspect kernel/networking related
things. sysctl -a showed some differences in various vm/numa/ipv4
settings. Simply changing settings around might be a good first step,
but if it's kernel related that might be tougher to track down. The
first step was to make sure it wasn't due to auth as that was suspect as
well, but it seems to be an independent issue.
Mark
On 02/06/2015 01:45 AM, Alexandre DERUMIER wrote:
Hi Mark,
do you known why rhel7 is faster than ubuntu ? (the difference seem to be quite huge)
Not sure it's related, but I have found a bug report on ubuntu,
about irqbalance not working correctly on ubuntu 14.04
https://bugs.launchpad.net/ubuntu/+source/irqbalance/+bug/1379065
----- Mail original -----
De: "Mark Nelson" <mark.nelson@xxxxxxxxxxx>
À: "Sage Weil" <sweil@xxxxxxxxxx>, "Andreas Bluemle" <andreas.bluemle@xxxxxxxxxxx>
Cc: "Stephen L Blinick" <stephen.l.blinick@xxxxxxxxx>, "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>
Envoyé: Jeudi 5 Février 2015 00:24:42
Objet: Re: wip-auth
Hi All,
I completed some tests with wip-auth to the memstore on ubuntu earlier
today. Basic gist of it is that the improvements in wip-auth help but
don't quite get us to what can be achieved with auth disabled. RHEL7
(without auth) continues to do very well in latency bound situations.
Next up will be to see if how much this matters when testing against on
SSDs.
Here are the results:
sync 4k object writes
=====================
Ceph OS Auth Avg Lat (ms) Avg IOPS
----------------------------------------------------------------
master Ubuntu 14.04 Yes 0.99 1007
Wip-auth Ubuntu 14.04 Yes 0.81 1237
master Ubuntu 14.04 No 0.64 1549
Wip-auth Ubuntu 14.04 No 0.65 1563
master RHEL7 No 0.32 3158
sync 4k object reads
====================
Ceph OS Auth Avg Lat (ms) Avg IOPS
----------------------------------------------------------------
master Ubuntu 14.04 Yes 0.59 1695
Wip-auth Ubuntu 14.04 Yes 0.41 2409
master Ubuntu 14.04 No 0.29 3425
Wip-auth Ubuntu 14.04 No 0.29 3474
master RHEL7 No 0.17 5853
256 concurrent 4k object writes
===============================
Ceph OS Auth Avg Lat (ms) Avg IOPS
----------------------------------------------------------------
master Ubuntu 14.04 Yes 40.39 6339
Wip-auth Ubuntu 14.04 Yes 26.22 9763
master Ubuntu 14.04 No 17.46 14662
Wip-auth Ubuntu 14.04 No 17.34 14759
master RHEL7 No 14.93 17139
256 concurrent 4k object reads
==============================
Ceph OS Auth Avg Lat (ms) Avg IOPS
----------------------------------------------------------------
master Ubuntu 14.04 Yes 31.47 8134
Wip-auth Ubuntu 14.04 Yes 19.81 12922
master Ubuntu 14.04 No 12.82 19968
Wip-auth Ubuntu 14.04 No 12.75 20080
master RHEL7 No 12.04 21257
Mark
On 01/30/2015 03:08 PM, Sage Weil wrote:
Hi Andreas,
It looks like that was a stale sha1, but the newer one was also
broken. I've retested and it's working for me now. See latest wip-auth,
sha1 0c21a7875059bef80842756dfb003f47cc2d66a6.
Thanks!
sage
On Fri, 30 Jan 2015, Andreas Bluemle wrote:
Hi Sage,
I tried to integrate wip-auth into my 0.91 build environment.
I had not been able to start the cluster successfully: ceph-mon
crashes with a segmentation fault in CryptoKey::encrypt()
(see attachment).
This happens when linking with libnss or libcryptopp (version 5.6.2).
I created the patch to add wip-auth based on github
pull request 3523 and was able to use this patch directly
with 0.91 with only a minor adaptation for common/ceph_context.h:
the 0.91 version of ceph_context.h did not know anything about
the experimental "class CephContextObs".
wip-auth commit ID is 1a0507a2940f6edcc2bf9533cfa6c210b0b41933.
As my build environment is rpm, I had to modify the invocation
of the configure script in the spec file instead of the do_autogen.sh
script.
Best Regards
Andreas Bluemle
On Tue, 27 Jan 2015 09:18:45 -0800 (PST)
Sage Weil <sweil@xxxxxxxxxx> wrote:
I spent some time focusing on just CryptoKey::encrypt(). I
benchmarked 100,000 encrypts of 128 bytes and got, at baseline,
cryptopp: 100000 encoded in 0.655651
libnss : 100000 encoded in 1.288786
Ouch! With a (fixed) version of my earlier patch that avoids
uselessly copying the input buffer:
100000 encoded in 1.231977
With a patch that puts the key structures in CryptoKey instead of
recreating them each time:
100000 encoded in 0.396208 -- ~70% improvement over original
This is pushed to wip-auth. There's also a patch that caches key
structs for crypopp.. it now takes
100000 encoded in 0.440758 -- ~33% improvement over original
(Not that almost anybody will ever care; we use libnss by default for
both rpm and deb distros.)
So, yay, nss is now a bit faster. What I'm not completely certain
about is whether the structures I've preserved are truly stateless
(and can be shared across threads, etc.). They encrypt/decrypt
methods are const so, if the libraries are const-correct, it should
be fine... but perhaps someone familiar with nss and/or crypto++ can
review this?
This is pushed to the latest wip-auth branch:
https://github.com/ceph/ceph/commits/wip-auth
Andreas and Stephen, what effect does this have on your numbers?
Thanks!
sage
On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
Good to know, I was wondering why the spec file defaulted to
lib-nss.. the dpkg-build for debian packages just uses whatever
configuration you had built, and I believe that will use
libcryptopp if the dependency is installed on the build machine
(last I looked).
I forgot to mention the numbers below were based on v.91.
Thanks,
Stephen
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx
[mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Sage Weil
Sent: Monday, January 26, 2015 10:24 AM To: Blinick, Stephen L
Cc: andreas.bluemle@xxxxxxxxxxx; ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: wip-auth
On Mon, 26 Jan 2015, Blinick, Stephen L wrote:
I noticed that the spec file for building RPM's defaults to
building with libnss, instead of libcrypto++. Since the
measurements I'd done so far were from those RPM's I rebuilt with
libcrypto++.. so FWIW here is the difference between those two on
my system, memstore backend with a single OSD, and single
client.
Dual socket Xeon E5 2620v3, 64GB Memory, RHEL7
Kernel: 3.10.0-123.13.2.el7
100% 4K Writes, 1xOSD w/ Rados Bench
libnss |
Cryptopp # QD IOPS Latency(ms) |
IOPS Latency(ms) IOPS Improvement % 16
14432.57 1.11 | 18896.60 0.85
30.93% 100% 4K Reads, 1xOSD w/ Rados
Bench libnss | Cryptopp # QD IOPS Latency(ms) | IOPS Latency(ms)
IOPS Improvement % 16 19532.53 0.82 | 25708.70 0.62 31.62%
Yikes, 30%! I think this definitely worth some effort. We
switched to libnss because it has the weird government
certfiications that everyone wants and is more prevalent. crypto++
is also not packaged for Red Hat distros at all (presumably for
that reason).
I suspect that most of the overhead is in the encryption context
setup and can be avoided with a bit of effort..
sage
Thanks,
Stephen
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx
[mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Sage Weil
Sent: Thursday, January 22, 2015 4:56 PM
To: andreas.bluemle@xxxxxxxxxxx
Cc: ceph-devel@xxxxxxxxxxxxxxx
Subject: wip-auth
Hi Andreas,
I took a look at the wip-auth I mentioned in the security call
last week... and the patch didn't work at all. Sorry if you
wasted any time trying it.
Anyway, I fixed it up so that it actually worked and made one
other optimization. It would be great to hear what latencies you
measure with the changes in place.
Also, it might be worth trying --with-cryptopp (or --with-nss if
you built cryptopp by default) to see if there is a difference.
There is a ton of boilerplate setting up encryption contexts and
key structures and so on that I suspect could be cached (perhaps
stashed in the CryptoKey struct?) with a bit of effort. See
https://github.com/ceph/ceph/blob/master/src/auth/Crypto.cc#L99-L213
sage
--
To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
-- To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Andreas Bluemle mailto:Andreas.Bluemle@xxxxxxxxxxx
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
