Earlier today I modified the rocksdb options so I could enable universal
compaction. Over all performance is lower but I don't see the
hang/stall in the middle of the test either. Instead the disk is
basically pegged with 100% writes. I suspect average latency is higher
than leveldb, but the highest latency is about 5-6s while we were seeing
30s spikes for leveldb with levelled (heh) compaction.
I haven't done much tuning either way yet. It may be that if we keep
level 0 and level 1 roughly the same size we can reduce stalls in the
levelled setups. It will also be interesting to see what happens in
these tests on SSDs.
Mark
On 07/24/2014 06:13 AM, Mark Nelson wrote:
Hi Xinxin,
Thanks! I wonder as well if it might be interesting to expose the
options related to universal compaction? It looks like rocksdb provides
a lot of interesting knobs you can adjust!
Mark
On 07/24/2014 12:08 AM, Shu, Xinxin wrote:
Hi mark,
I think this maybe related to 'verify_checksums' config option ,when
ReadOptions is initialized, default this option is true , all data
read from underlying storage will be verified against corresponding
checksums, however, this option cannot be configured in wip-rocksdb
branch. I will modify code to make this option configurable .
Cheers,
xinxin
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx
[mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Mark Nelson
Sent: Thursday, July 24, 2014 7:14 AM
To: ceph-devel@xxxxxxxxxxxxxxx
Subject: First attempt at rocksdb monitor store stress testing
Hi Guys,
So I've been interested lately in leveldb 99th percentile latency (and
the amount of write amplification we are seeing) with leveldb. Joao
mentioned he has written a tool called mon-store-stress in
wip-leveldb-misc to try to provide a means to roughly guess at what's
happening on the mons under heavy load. I cherry-picked it over to
wip-rocksdb and after a couple of hacks was able to get everything
built and running with some basic tests. There was little tuning done
and I don't know how realistic this workload is, so don't assume this
means anything yet, but some initial results are here:
http://nhm.ceph.com/mon-store-stress/First%20Attempt.pdf
Command that was used to run the tests:
./ceph-test-mon-store-stress --mon-keyvaluedb <leveldb|rocksdb>
--write-min-size 50K --write-max-size 2M --percent-write 70
--percent-read 30 --keep-state --test-seed 1406137270 --stop-at 5000 foo
The most interesting bit right now is that rocksdb seems to be hanging
in the middle of the test (left it running for several hours). CPU
usage on one core was quite high during the hang. Profiling using
perf with dwarf symbols I see:
- 49.14% ceph-test-mon-s ceph-test-mon-store-stress [.] unsigned
int rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
int, char const*, unsigned long)
- unsigned int
rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
int, char const*, unsigned long)
51.70% rocksdb::ReadBlockContents(rocksdb::RandomAccessFile*,
rocksdb::Footer const&, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::Env*,
bool)
48.30%
rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&,
rocksdb::CompressionType, rocksdb::BlockHandle*)
Not sure what's going on yet, may need to try to enable
logging/debugging in rocksdb. Thoughts/Suggestions welcome. :)
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo
info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
