On 3/12/19 7:31 AM, vitalif@xxxxxxxxxx wrote:
Decreasing the min_alloc size isn't always a win, but ican be in some
cases. Originally bluestore_min_alloc_size_ssd was set to 4096 but we
increased it to 16384 because at the time our metadata path was slow
and increasing it resulted in a pretty significant performance win
(along with increasing the WAL buffers in rocksdb to reduce write
amplification). Since then we've improved the metadata path to the
point where at least on our test nodes performance is pretty close
between with min_alloc size = 16k and min_alloc size = 4k the last
time I looked. It might be a good idea to drop it down to 4k now but
I think we need to be careful because there are tradeoffs.
I think it's all about your disks' latency. Deferred write is 1
IO+sync and redirect-write is 2 IOs+syncs. So if your IO or sync is
slow (like it is on HDDs and bad SSDs) then the deferred write is
better in terms of latency. If your IO is fast then you're only
bottlenecked by the OSD code itself eating a lot of CPU and then
direct write may be better. By the way, I think OSD itself is way TOO
slow currently (see below).
Don't disagree, bluestore's write path has gotten *really* complicated.
The idea I was talking about turned out to be only viable for HDD/slow
SSDs and only for low iodepths. But the gain is huge - something
between +50% iops to +100% iops (2x less latency). There is a stupid
problem in current bluestore implementation which makes it do 2
journal writes and FSYNCs instead of one for every incoming
transaction. The details are here: https://tracker.ceph.com/issues/38559
The unnecessary commit is the BlueFS's WAL. All it's doing is
recording the increased size of a RocksDB WAL file. Which obviously
shouldn't be required with RocksDB as its default setting is
"kTolerateCorruptedTailRecords". However, without this setting the WAL
is not synced to the disk with every write because by some clever
logic sync_file_range is called only with SYNC_FILE_RANGE_WRITE in the
corresponding piece of code. Thus the OSD's database gets corrupted
when you kill it with -9 and thus it's impossible to set
`bluefs_preextend_wal_files` to true. And thus you get two writes and
commits instead of one.
I don't know the exact idea behind doing only SYNC_FILE_RANGE_WRITE -
as I understand there is currently no benefit in doing this. It could
be a benefit if RocksDB was writing journal in small parts and then
doing a single sync - but it's always flushing the newly written part
of a journal to disk as a whole.
The simplest way to fix it is just to add SYNC_FILE_RANGE_WAIT_BEFORE
and SYNC_FILE_RANGE_WAIT_AFTER to sync_file_range in KernelDevice.cc.
My pull request is here: https://github.com/ceph/ceph/pull/26909 -
I've tested this change with 13.2.4 Mimic and 14.1.0 Nautilus and yes,
it does increase single-thread iops on HDDs two times (!). After this
change BlueStore becomes actually better than FileStore at least on HDDs.
Another way of fixing it would be to add an explicit bdev->flush at
the end of the kv_sync_thread, after db->submit_transaction_sync(),
and possibly remove the redundant sync_file_range at all. But then you
must do the same in another place in _txc_state_proc, because it's
also sometimes doing submit_transaction_sync(). In the end I
personally think that to add flags to sync_file_range is better
because a function named "submit_transaction_sync" should be in fact
SYNC! It shouldn't require additional steps from the caller to make
the data durable.
I'm glad you are peaking under the covers here. :) There's a lot going
on here, and it's not immediate obvious what the intent is and the
failure conditions are. I suspect the intent here was to error on the
side of caution but we really need to document this better. To be fair
it's not just us, there's confusion and terribleness all the way up to
the kernel and beyond.
Also I have a small funny test result to share.
I've created one OSD on my laptop on a loop device in a tmpfs (i.e.
RAM), created 1 RBD image inside it and tested it with `fio
-ioengine=rbd -direct=1 -bs=4k -rw=randwrite`. Before doing the test
I've turned off CPU power saving with `cpupower idle-set -D 0`.
The results are:
- filestore: 2200 iops with -iodepth=1 (0.454ms average latency). 8500
iops with -iodepth=128.
- bluestore: 1800 iops with -iodepth=1 (0.555ms average latency). 9000
iops with -iodepth=128.
- memstore: 3000 iops with -iodepth=1 (0.333ms average latency). 11000
iops with -iodepth=128.
If we can think of memstore being a "minimal possible /dev/null" then:
- OSD overhead is 1/3000 = 0.333ms (maybe slighly less, but that
doesn't matter).
- filestore overhead is 1/2200-1/3000 = 0.121ms
- bluestore overhead is 1/1800-1/3000 = 0.222ms
The conclusion is that bluestore is actually almost TWO TIMES slower
than filestore in terms of pure latency, and the throughput is only
slightly better. How could it happen? How could a newly written store
become two times slower than the old one? ) that's pretty annoying...
I bet you'd see better memstore results with my vector based object
implementation instead of bufferlists. Nick Fisk noticed the same thing
you did. One interesting observation he made was that disabling CPU C/P
states helped bluestore immensely in the iodepth=1 case. IE, bluestore
does so much in it's write path that it's really sensitive to latency
introduced by C state transitions. Just more fodder showing that the
bluestore write path is really complicated. I think bluestore was still
the right way to go vs filestore (for a variety of reasons!) but I think
there would be significant benefit to auditing the write path.
Could it be because bluestore is doing a lot of threading? I mean
could it be because each write operation goes through 5 threads during
its execution? (tp_osd_tp -> aio -> kv_sync_thread ->
kv_finalize_thread -> finisher)? Maybe just remove aio and kv threads
and process all operations directly in tp_osd_tp then?
One way or another we can only have a single thread sending writes to
rocksdb. A lot of the prior optimization work on the write side was to
get as much processing out of the kv_sync_thread as possible. That's
still a worthwhile goal as it's typically what bottlenecks with high
amounts of concurrency. What I think would be very interesting though
is if we moved more toward a model where we had lots shards (OSDs or
shards of an OSD) with independent rocksdb instances and less threading
overhead per shard. That's the way the seastar work is going, and also
sort of the model I've been thinking about for a very simple
single-threaded OSD.
Mark
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
