On 12/23/2016 01:24 PM, Allen Samuels wrote:
Be careful assuming that 100/200 is the permanent "right" setting.
See what I wrote:
"It looks like 100/200 is probably the current optimal configuration on
my test setup."
I do choose my words carefully Allen. :)
Mark
If all you're doing is random IOPS, then the best setting ought to be
"1". What you're seeing is diminishing returns, which can only be seen
in context of the full code-path. Thus if there's a significant
acceleration of the other parts of the code path, then you'll find that
reducing the 100/200 will again yield significant performance benfits.
Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, San Jose, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@xxxxxxxxxxx
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-
owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson
Sent: Friday, December 23, 2016 9:09 AM
To: Sage Weil <sweil@xxxxxxxxxx>
Cc: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>; ceph-devel <ceph-
devel@xxxxxxxxxxxxxxx>
Subject: Re: Bluestore performance bottleneck
Try this?
https://github.com/ceph/ceph/pull/12634
Looks like this is most likely reducing the memory usage and
increasing performance quite a bit with smaller shard target/max
values. With
25/50 I'm seeing more like 2.6GB RSS memory usage and around 13K iops
typically with some (likely rocksdb) stalls. I'll run through the
tests again.
Mark
Ok, Ran through tests with both 4k and 16k min_alloc/max_alloc/blob sizes
using master+12629+12634:
https://drive.google.com/uc?export=download&id=0B2gTBZrkrnpZQzdRU3B
1SGZUbDQ
Performance is up in all tests and memory consumption is down (especially in
the smaller target/max tests). It looks like 100/200 is probably the current
optimal configuration on my test setup. 4K min_alloc tests hover around
22.5K IOPS with ~1300% CPU usage, and 16K min_alloc tests hover around
25K IOPs with ~1000% CPU usage. I think it will be worth spending some time
looking at locking in the bitmap allocator given the perf traces. Beyond that,
I'm seeing rocksdb show up quite a bit in the top CPU consuming functions
now, especially CRC32.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
