Yes you did. I am properly chastised. Allen Samuels SanDisk |a Western Digital brand 2880 Junction Avenue, San Jose, CA 95134 T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > -----Original Message----- > From: Mark Nelson [mailto:mnelson@xxxxxxxxxx] > Sent: Friday, December 23, 2016 11:33 AM > To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Sage Weil > <sweil@xxxxxxxxxx> > Cc: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>; ceph-devel <ceph- > devel@xxxxxxxxxxxxxxx> > Subject: Re: Bluestore performance bottleneck > > On 12/23/2016 01:24 PM, Allen Samuels wrote: > > Be careful assuming that 100/200 is the permanent "right" setting. > > See what I wrote: > > "It looks like 100/200 is probably the current optimal configuration on my test > setup." > > I do choose my words carefully Allen. :) > > Mark > > If all you're doing is random IOPS, then the best setting ought to be "1". What > you're seeing is diminishing returns, which can only be seen in context of the > full code-path. Thus if there's a significant acceleration of the other parts of > the code path, then you'll find that reducing the 100/200 will again yield > significant performance benfits. > > > > > > Allen Samuels > > SanDisk |a Western Digital brand > > 2880 Junction Avenue, San Jose, CA 95134 > > T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx > > > > > >> -----Original Message----- > >> From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel- > >> owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson > >> Sent: Friday, December 23, 2016 9:09 AM > >> To: Sage Weil <sweil@xxxxxxxxxx> > >> Cc: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>; ceph-devel <ceph- > >> devel@xxxxxxxxxxxxxxx> > >> Subject: Re: Bluestore performance bottleneck > >> > >>>> Try this? > >>>> https://github.com/ceph/ceph/pull/12634 > >>> > >>> Looks like this is most likely reducing the memory usage and > >>> increasing performance quite a bit with smaller shard target/max > >>> values. With > >>> 25/50 I'm seeing more like 2.6GB RSS memory usage and around 13K iops > >>> typically with some (likely rocksdb) stalls. I'll run through the > >>> tests again. > >>> > >>> Mark > >>> > >> > >> Ok, Ran through tests with both 4k and 16k min_alloc/max_alloc/blob > sizes > >> using master+12629+12634: > >> > >> > https://drive.google.com/uc?export=download&id=0B2gTBZrkrnpZQzdRU3B > >> 1SGZUbDQ > >> > >> Performance is up in all tests and memory consumption is down > (especially in > >> the smaller target/max tests). It looks like 100/200 is probably the current > >> optimal configuration on my test setup. 4K min_alloc tests hover around > >> 22.5K IOPS with ~1300% CPU usage, and 16K min_alloc tests hover around > >> 25K IOPs with ~1000% CPU usage. I think it will be worth spending some > time > >> looking at locking in the bitmap allocator given the perf traces. Beyond > that, > >> I'm seeing rocksdb show up quite a bit in the top CPU consuming > functions > >> now, especially CRC32. > >> > >> Mark > >> > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the > >> body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info > at > >> http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
