On Thu, 18 Aug 2016, Somnath Roy wrote: > Sage, Here is some estimate probably how much extra (space wise) we are > writing with Bluestore. Considering Rocksdb has not much space amp for > level style compaction , these are mostly the metadata Bluestore is > writing. > > BlueStore: > ---------------- > root@stormeap-1:~/fio_rbd/fio/examples# ceph df > 2016-08-17 11:00:57.936952 7f5c39a2b700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > 2016-08-17 11:00:57.939969 7f5c39a2b700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > 2016-08-17 11:00:57.941269 7f5c39a2b700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 106T 22411G 86867G 79.49 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > recovery_test 1 39062G 71.49 6290G 10000003 > > > So, if we trust the statfs implementation of Bluestore , it is writing ~8743 GB more. Total data = image size of 39062 GB * replication 2 = ~78124 GB. So, ~11.19% more. > BTW, this is after 1MB image preconditioning only, filling with 4K blocksize will be adding more metadata. > > Filestore: > ----------- > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 109T 34443G 78147G 69.41 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > recovery_test 2 39062G 69.39 12930G 10000003 > > So, in the similar setup , filestore is writing only ~23GB extra and i.e ~0.029% This seems like a lot for bluestore. The statfs output from bluestore should show how much of the space is bluefs vs bluestore. Hmm, my guess is that bluestore is counting all of the space that it has given to bluefs as used, even though bluefs isn't using it. Probably just need to make BlueStore::statfs() call BlueFs::statfs() and correct for the bluefs unused space... sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
