Interesting, I see all of the usual suspects here with InlineSkipList
KeyComparator being the big one. I've rarely seen it this bad though.
What model CPU are you running on?
There's a very good chance that you would benefit from the new
(experimental) tuning in the RocksDB article. Smaller buffers means
fewer key comparisons and that's what's primarily holding you back based
on this trace. The trade off used to be higher write-amplification but
making sure that L0 and L1 are properly sized seems to help avoid that.
We can't support that tuning yet, but if you have a test cluster it
might be something worth trying out.
Mark
On 11/10/22 10:54, Eshcar Hillel wrote:
I attach the profiling results of one of the OSDs
you are invited to take a look at it.
we already have multiple OSDs on a single nvme drive.
we have 3 nodes each with 4 SSD drives.
We run 8 SSDs on each nvme, total of 96 OSDs.
Indeed, the size of the data in each rocksdb db is very small and the
reported compaction WA is ~1.4 for each cf.
My suggestion is not to shard across KeyValueDBs.
instead, I suggest having multiple bstore_kv_sync accessing the same
cfs concurrently.
rocksdb has a feature that allows concurrent writes:
https://www.youtube.com/watch?v=_OBCvU-DECk
it is disabled by default, but it is there.
the assumption is that due to internal sharding most operations would
access different cfs and in practice there will be no conflicts and so
no concurrency overhead.
nevertheless, the concurrent write flag will allow those operation
that do need to share the same cf to do it in a safe way.
was this option ever considered?
------------------------------------------------------------------------
*From:* Mark Nelson <mnelson@xxxxxxxxxx>
*Sent:* Wednesday, November 9, 2022 3:09 PM
*To:* Eshcar Hillel <eshcarh@xxxxxxxxxx>; ceph-users@xxxxxxx
<ceph-users@xxxxxxx>
*Subject:* Re: Recent ceph.io Performance Blog Posts
CAUTION:External Sender
On 11/9/22 6:03 AM, Eshcar Hillel wrote:
Hi Mark,
Thanks for posting these blogs. They are very interesting to read.
Maybe you have an answer to a question I asked in the dev list:
We run fio benchmark against a 3-node ceph cluster with 96 OSDs.
Objects are 4kb. We use
gdbpmp profilerhttps://github.com/markhpc/gdbpmp
<https://github.com/markhpc/gdbpmp> to analyze the threads' performance.
we discovered the bstore_kv_sync thread is always busy, while all 16
tp_osd_tp threads are not busy most of the time (wait on a
conditional variable or a lock).
Given that 3 rocksdb CFs are sharded, and sharding is configurable,
why not run multiple (3) bstore_kv_sync threads? they won't have
conflicts most of the time.
This has the potential of removing the rocksdb bottleneck and
increasing IOPS.
Can you explain this design choice?
You are absolutely correct that the bstore_kv_sync thread can often be
a bottleneck during 4K random writes. Typically it's not so bad that
the tp_osd_tp threads are mostly blocked though (feel free to send me
a copy of the trace, I would be interested in seeing it). Years ago I
advocated for the same approach you are suggesting here. The fear at
the time was that the changes inside bluestore would be too
disruptive. The column family sharding approach could be (and was)
mostly contained to the KeyValueDB glue code. Column family sharding
has been a win from the standpoint that it helps us avoid really deep
LSM hierarchies in RocksDB. We tend to see faster compaction times
and are more likely to keep full levels on the fast device. Sadly it
doesn't really help with improving metadata throughput and may even
introduce a small amount of overhead during the WAL flush process.
FWIW slow bstore_kv_sync is one of the reasons that people will some
times run multiple OSDs on one NVMe drive (sometimes it's faster,
sometimes it's not).
Maybe a year ago I tried to sort of map out the changes that I thought
would be necessary to shard across KeyValueDBs inside bluestore
itself. It didn't look impossible, but would require quite a bit of
work (and a bit of finesse to restructure the data path). There's a
legitimate questions of whether or not it's worth it now to make those
kinds of changes to bluestore or invest in crimson and seastore at
this point. We ended up deciding not to pursue the changes back
then. I think if we changed our minds it would most likely go into
some kind of experimental bluestore v2 project (along with other
things like hierarchical storage) so we don't screw up the existing
code base.
------------------------------------------------------------------------
*From:* Mark Nelson <mnelson@xxxxxxxxxx> <mailto:mnelson@xxxxxxxxxx>
*Sent:* Tuesday, November 8, 2022 10:20 PM
*To:* ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
<ceph-users@xxxxxxx> <mailto:ceph-users@xxxxxxx>
*Subject:* Recent ceph.io Performance Blog Posts
CAUTION: External Sender
Hi Folks,
I thought I would mention that I've released a couple of performance
articles on the Ceph blog recently that might be of interest to people:
1.
https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/
<https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/>
<https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/
<https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/>>
2.
https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/
<https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/>
<https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/
<https://ceph.io/en/news/blog/2022/qemu-kvm-tuning/>>
3.
https://ceph.io/en/news/blog/2022/ceph-osd-cpu-scaling/
<https://ceph.io/en/news/blog/2022/ceph-osd-cpu-scaling/>
<https://ceph.io/en/news/blog/2022/ceph-osd-cpu-scaling/
<https://ceph.io/en/news/blog/2022/ceph-osd-cpu-scaling/>>
The first covers RocksDB tuning. How we arrived at our defaults, an
analysis of some common settings that have been floating around on the
mailing list, and potential new settings that we are considering making
default in the future.
The second covers how to tune QEMU/KVM with librbd to achieve high
single-client performance on a small (30 OSD) NVMe backed cluster. This
article also covers the performance impact when enabling 128bit AES
over-the-wire encryption.
The third covers per-OSD CPU/Core scaling and the kind of IOPS/core and
IOPS/NVMe numbers that are achievable both on a single OSD and on a
larger (60 OSD) NVMe cluster. In this case there are enough clients and
a high enough per-client iodepth to saturate the OSD(s).
I hope these are helpful or at least interesting!
Thanks,
Mark
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx
<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
