-----Original Message-----
From: Igor Fedotov [mailto:ifedotov@xxxxxxxxxxxx]
Sent: Friday, June 10, 2016 9:51 AM
To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Sage Weil
<sweil@xxxxxxxxxx>; Somnath Roy <Somnath.Roy@xxxxxxxxxxx>
Cc: Mark Nelson <mnelson@xxxxxxxxxx>; Manavalan Krishnan
<Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph-
devel@xxxxxxxxxxxxxxx>
Subject: Re: RocksDB tuning
An update:
I found that my previous results were invalid - SyntheticWorkloadState had
an odd swap for offset > len case... Made a brief fix.
Now onode size with csum raises up to 38K, without csum - 28K.
For csum case there is 350 lextents and about 170 blobs
For no csum - 343 lextents and about 170 blobs.
(blobs counting is very inaccurate!)
Potentially we shouldn't have >64 blobs per 4M thus looks like some issues in
the write path...
And CSum vs. NoCsum differenct looks pretty consistent - 170 blobs * 4 byte
* 16 values = 10880
Branch's @github been updated with corresponding fixes.
Thanks,
Igor.
On 10.06.2016 19:06, Allen Samuels wrote:
Let's see, 4MB is 2^22 bytes. If we storage a checksum for each 2^12 bytes
that's 2^10 checksums at 2^2 bytes each is 2^12 bytes.
So with optimal encoding, the checksum baggage shouldn't be more than
4KB per oNode.
But you're seeing 13K as the upper bound on the onode size.
In the worst case, you'll need at least another block address (8 bytes
currently) and length (another 8 bytes) [though as I point out, the length is
something that can be optimized out] So worst case, this encoding would be
an addition 16KB per onode.
I suspect you're not at the worst-case yet :)
Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, Milpitas, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx
-----Original Message-----
From: Igor Fedotov [mailto:ifedotov@xxxxxxxxxxxx]
Sent: Friday, June 10, 2016 8:58 AM
To: Sage Weil <sweil@xxxxxxxxxx>; Somnath Roy
<Somnath.Roy@xxxxxxxxxxx>
Cc: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>; Mark Nelson
<mnelson@xxxxxxxxxx>; Manavalan Krishnan
<Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph-
devel@xxxxxxxxxxxxxxx>
Subject: Re: RocksDB tuning
Just modified store_test synthetic test case to simulate many random 4K
writes to 4M object.
With default settings ( crc32c + 4K block) onode size varies from 2K to
~13K
with disabled crc it's ~500 - 1300 bytes.
Hence the root cause seems to be in csum array.
Here is the updated branch:
https://github.com/ifed01/ceph/tree/wip-bluestore-test-size
Thanks,
Igor
On 10.06.2016 18:40, Sage Weil wrote:
On Fri, 10 Jun 2016, Somnath Roy wrote:
Just turning off checksum with the below param is not helping, I
still need to see the onode size though by enabling debug..Do I need
to mkfs
(Sage?) as it is still holding checksum of old data I wrote ?
Yeah.. you'll need to mkfs to blow away the old onodes and blobs with
csum data.
As Allen pointed out, this is only part of the problem.. but I'm
curious how much!
bluestore_csum = false
bluestore_csum_type = none
Here is the snippet of 'dstat'..
----total-cpu-usage---- -dsk/total- -net/total- ---paging-->
usr sys idl wai hiq siq| read writ| recv send| in out >
41 14 36 5 0 4| 138M 841M| 212M 145M| 0 0 >
42 14 35 5 0 4| 137M 855M| 213M 147M| 0 0 >
40 14 38 5 0 3| 143M 815M| 209M 144M| 0 0 >
40 14 38 5 0 3| 137M 933M| 194M 134M| 0 0 >
42 15 34 5 0 4| 133M 918M| 220M 151M| 0 0 >
35 13 43 6 0 3| 147M 788M| 194M 134M| 0 0 >
31 11 49 6 0 3| 157M 713M| 151M 104M| 0 0 >
39 14 38 5 0 4| 139M 836M| 246M 169M| 0 0 >
40 14 38 5 0 3| 139M 845M| 204M 140M| 0 0 >
40 14 37 5 0 4| 149M 743M| 210M 144M| 0 0 >
42 14 35 5 0 4| 143M 852M| 216M 150M| 0 0 >
For example, what last entry is saying that system (with 8 osds) is
receiving 216M of data over network and in response to that it is writing
total
of 852M of data and reading 143M of data. At this time FIO on client side is
reporting ~35K 4K RW iops.
Now, after a min or so, the throughput goes down to barely 1K from
FIO
(and very bumpy) and here is the 'dstat' snippet at that time..
----total-cpu-usage---- -dsk/total- -net/total- ---paging-->
usr sys idl wai hiq siq| read writ| recv send| in out >
2 1 83 14 0 0| 220M 58M|4346k 3002k| 0 0 >
2 1 82 14 0 0| 223M 60M|4050k 2919k| 0 0 >
3 1 82 13 0 0| 217M 63M|6403k 4306k| 0 0 >
2 1 83 14 0 0| 226M 54M|2126k 1497k| 0 0 >
So, system is barely receiving anything (~2M) but still writing ~54M of
data
and reading 226M of data from disk.
After killing fio script , here is the 'dstat' output..
----total-cpu-usage---- -dsk/total- -net/total- ---paging-->
usr sys idl wai hiq siq| read writ| recv send| in out >
2 1 86 12 0 0| 186M 66M| 28k 26k| 0 0 >
2 1 86 12 0 0| 201M 78M| 20k 21k| 0 0 >
2 1 85 12 0 0| 230M 100M| 24k 24k| 0 0 >
2 1 85 12 0 0| 206M 78M| 21k 20k| 0 0 >
Not receiving anything from client but still writing 78M of data and
206M
of read.
Clearly, it is an effect of rocksdb compaction that stalling IO and even if
we
increased compaction thread (and other tuning), compaction is not able to
keep up with incoming IO.
Thanks & Regards
Somnath
-----Original Message-----
From: Allen Samuels
Sent: Friday, June 10, 2016 8:06 AM
To: Sage Weil
Cc: Somnath Roy; Mark Nelson; Manavalan Krishnan; Ceph
Development
Subject: RE: RocksDB tuning
-----Original Message-----
From: Sage Weil [mailto:sweil@xxxxxxxxxx]
Sent: Friday, June 10, 2016 7:55 AM
To: Allen Samuels <Allen.Samuels@xxxxxxxxxxx>
Cc: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>; Mark Nelson
<mnelson@xxxxxxxxxx>; Manavalan Krishnan
<Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph-
devel@xxxxxxxxxxxxxxx>
Subject: RE: RocksDB tuning
On Fri, 10 Jun 2016, Allen Samuels wrote:
Checksums are definitely a part of the problem, but I suspect the
smaller part of the problem. This particular use-case (random 4K
overwrites without the WAL stuff) is the worst-case from an
encoding perspective and highlights the inefficiency in the current
code.
As has been discussed earlier, a specialized encode/decode
implementation for these data structures is clearly called for.
IMO, you'll be able to cut the size of this by AT LEAST a factor of
3 or
4 without a lot of effort. The price will be somewhat increase CPU
cost for the serialize/deserialize operation.
If you think of this as an application-specific data compression
problem, here is a short list of potential compression opportunities.
(1) Encoded sizes and offsets are 8-byte byte values, converting
these too
block values will drop 9 or 12 bits from each value. Also, the
ranges for these values is usually only 2^22 -- often much less.
Meaning that there's 3-5 bytes of zeros at the top of each word that
can
be dropped.
(2) Encoded device addresses are often less than 2^32, meaning
there's 3-4
bytes of zeros at the top of each word that can be dropped.
(3) Encoded offsets and sizes are often exactly "1" block, clever
choices of
formatting can eliminate these entirely.
IMO, an optimized encoded form of the extent table will be around
1/4 of the current encoding (for this use-case) and will likely
result in an Onode that's only 1/3 of the size that Somnath is seeing.
That will be true for the lextent and blob extent maps. I'm
guessing this is a small part of the ~5K somnath saw. If his
objects are 4MB then 4KB of it
(80%) is the csum_data vector, which is a flat vector of
u32 values that are presumably not very compressible.
I don't think that's what Somnath is seeing (obviously some data here
will
sharpen up our speculations). But in his use case, I believe that he has a
separate blob and pextent for each 4K write (since it's been subjected to
random 4K overwrites), that means somewhere in the data structures at
least one address and one length for each of the 4K blocks (and likely
much
more in the lextent and blob maps as you alluded to above). The encoding
of
just this information alone is larger than the checksum data.
We could perhaps break these into a separate key or keyspace..
That'll give rocksdb a bit more computation work to do (for a custom
merge operator, probably, to update just a piece of the value) but
for a 4KB value I'm not sure it's big enough to really help. Also
we'd lose locality, would need a second get to load csum metadata on
read, etc.
:/ I don't really have any good ideas here.
sage
Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, Milpitas, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416 allen.samuels@xxxxxxxxxxx
-----Original Message-----
From: Sage Weil [mailto:sweil@xxxxxxxxxx]
Sent: Friday, June 10, 2016 2:35 AM
To: Somnath Roy <Somnath.Roy@xxxxxxxxxxx>
Cc: Mark Nelson <mnelson@xxxxxxxxxx>; Allen Samuels
<Allen.Samuels@xxxxxxxxxxx>; Manavalan Krishnan
<Manavalan.Krishnan@xxxxxxxxxxx>; Ceph Development <ceph-
devel@xxxxxxxxxxxxxxx>
Subject: RE: RocksDB tuning
On Fri, 10 Jun 2016, Somnath Roy wrote:
Sage/Mark,
I debugged the code and it seems there is no WAL write going on
and
working as expected. But, in the process, I found that onode size
it is
writing
to my environment ~7K !! See this debug print.
2016-06-09 15:49:24.710149 7f7732fe3700 20
bluestore(/var/lib/ceph/osd/ceph-0) onode
#1:7d3c6423:::rbd_data.10186b8b4567.0000000000070cd4:head# is
7518
This explains why so much data going to rocksdb I guess. Once
compaction kicks in iops I am getting is *30 times* slower.
I have 15 osds on 8TB drives and I have created 4TB rbd image
preconditioned with 1M. I was running 4K RW test.
The onode is big because of the csum metdata. Try setting
'bluestore
csum
type = none' and see if that is the entire reason or if something
else is
going
on.
We may need to reconsider the way this is stored.
s
Thanks & Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx
[mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of
Somnath
Roy
Sent: Thursday, June 09, 2016 8:23 AM
To: Mark Nelson; Allen Samuels; Manavalan Krishnan; Ceph
Development
Subject: RE: RocksDB tuning
Mark,
As we discussed, it seems there is ~5X write amp on the system
with 4K
RW. Considering the amount of data going into rocksdb (and thus
kicking
of
compaction so fast and degrading performance drastically) , it
seems it is
still
writing WAL (?)..I used the following rocksdb option for faster
background
compaction as well hoping it can keep up with upcoming writes and
writes
won't be stalling. But, eventually, after a min or so, it is stalling io..
bluestore_rocksdb_options =
"compression=kNoCompression,max_write_buffer_number=16,min_write_
buffer_number_to_merge=3,recycle_log_file_num=16,compaction_style=k
CompactionStyleLevel,write_buffer_size=67108864,target_file_size_bas
e=6
7108864,max_background_compactions=31,level0_file_num_compaction_tri
gger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=
64,
num_levels=4,max_bytes_for_level_base=536870912,max_bytes_for_level
_multiplier=8,compaction_threads=32,flusher_threads=8"
I will try to debug what is going on there..
Thanks & Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx
[mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark
Nelson
Sent: Thursday, June 09, 2016 6:46 AM
To: Allen Samuels; Manavalan Krishnan; Ceph Development
Subject: Re: RocksDB tuning
On 06/09/2016 08:37 AM, Mark Nelson wrote:
Hi Allen,
On a somewhat related note, I wanted to mention that I had
forgotten
that chhabaremesh's min_alloc_size commit for different media
types was committed into master:
https://github.com/ceph/ceph/commit/8185f2d356911274ca679614611dc335
e3
efd187
IE those tests appear to already have been using a 4K min alloc
size due to non-rotational NVMe media. I went back and
verified
that explicitly changing the min_alloc size (in fact all of them
to be
sure) to 4k does not change the behavior from graphs I showed
yesterday. The rocksdb compaction stalls due to excessive reads
appear (at least on the
surface) to be due to metadata traffic during heavy small
random
writes.
Sorry, this was worded poorly. Traffic due to compaction of
metadata
(ie
not leaked WAL data) during small random writes.
Mark
Mark
On 06/08/2016 06:52 PM, Allen Samuels wrote:
Let's make a patch that creates actual Ceph parameters for
these things so that we don't have to edit the source code in
the
future.
Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, San Jose, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@xxxxxxxxxxx
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-
devel-
owner@xxxxxxxxxxxxxxx] On Behalf Of Manavalan Krishnan
Sent: Wednesday, June 08, 2016 3:10 PM
To: Mark Nelson <mnelson@xxxxxxxxxx>; Ceph
Development
<ceph-
devel@xxxxxxxxxxxxxxx>
Subject: RocksDB tuning
Hi Mark
Here are the tunings that we used to avoid the IOPs
choppiness
caused by rocksdb compaction.
We need to add the following options in
src/kv/RocksDBStore.cc
before rocksdb::DB::Open in RocksDBStore::do_open
opt.IncreaseParallelism(16);
opt.OptimizeLevelStyleCompaction(512 * 1024 * 1024);
Thanks
Mana
PLEASE NOTE: The information contained in this electronic
mail
message is intended only for the use of the designated
recipient(s) named above.
If the
reader of this message is not the intended recipient, you are
hereby notified that you have received this message in error
and that any review, dissemination, distribution, or copying
of this message is strictly prohibited. If you have received
this communication in error, please notify the sender by
telephone or e-mail (as shown
above) immediately and destroy any and all copies of this
message in your possession (whether hard copies or
electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe
ceph-
devel"
in the
body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info
at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo info at http://vger.kernel.org/majordomo-
info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info at http://vger.kernel.org/majordomo-info.html
PLEASE NOTE: The information contained in this electronic mail
message
is
intended only for the use of the designated recipient(s) named
above. If
the
reader of this message is not the intended recipient, you are
hereby
notified
that you have received this message in error and that any review,
dissemination, distribution, or copying of this message is
strictly
prohibited. If
you have received this communication in error, please notify the
sender
by
telephone or e-mail (as shown above) immediately and destroy
any
and
all
copies of this message in your possession (whether hard copies or
electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-
devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe
ceph-devel" in the body of a message to
majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-
info.html
PLEASE NOTE: The information contained in this electronic mail
message is
intended only for the use of the designated recipient(s) named above. If
the
reader of this message is not the intended recipient, you are hereby
notified
that you have received this message in error and that any review,
dissemination, distribution, or copying of this message is strictly
prohibited. If
you have received this communication in error, please notify the sender
by
telephone or e-mail (as shown above) immediately and destroy any and
all
copies of this message in your possession (whether hard copies or
electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More
majordomo
info at http://vger.kernel.org/majordomo-info.html
