Re: BlueStore write amplification

[Jan Schermer <jan@xxxxxxxxxxx>]

Is that 400MB on all nodes or on each node? If it's on all nodes then 10:1 is not that surprising.

What what the block size in your fio benchmark?

We had much higher amplification on our cluster with snapshots and stuff...

Jan

On 23 Aug 2016, at 08:38, Zhiyuan Wang <zhiyuan.wang@xxxxxxxxxxx> wrote:

Hi 

I have test bluestore on SSD, and I found that the BW from fio is about 40MB, but

the write BW from iostat of SSD is about 400MB, nearly ten times.

Could someone help to explain this? 

Thanks a lot.

 

Below are my configuration file:

[global]

        fsid = 31e77e3c-447c-4745-a91a-58bda80a868c

        enable experimental unrecoverable data corrupting features = bluestore rocksdb

        osd objectstore = bluestore

 

        bluestore default buffered read = true

        bluestore_min_alloc_size=4096

        osd pool default size = 1

 

        osd pg bits = 8

        osd pgp bits = 8

        auth supported = none

        log to syslog = false

        filestore xattr use omap = true

        auth cluster required = none

        auth service required = none

        auth client required = none

 

        public network = 192.168.200.233/24

        cluster network = 192.168.100.233/24

 

        mon initial members = node3

        mon host = 192.168.200.233

        mon data = "" class="">

          

        filestore merge threshold = 40

        filestore split multiple = 8

        osd op threads = 8

             

        debug_bluefs = "0/0"

        debug_bluestore = "0/0"

        debug_bdev = "0/0" 

        debug_lockdep = "0/0" 

        debug_context = "0/0"  

        debug_crush = "0/0"

        debug_mds = "0/0"

        debug_mds_balancer = "0/0"

        debug_mds_locker = "0/0"

        debug_mds_log = "0/0"

        debug_mds_log_expire = "0/0"

        debug_mds_migrator = "0/0"

        debug_buffer = "0/0"

        debug_timer = "0/0"

        debug_filer = "0/0"

        debug_objecter = "0/0"

        debug_rados = "0/0"

        debug_rbd = "0/0"

        debug_journaler = "0/0"

        debug_objectcacher = "0/0"

        debug_client = "0/0"

        debug_osd = "0/0"

        debug_optracker = "0/0"

        debug_objclass = "0/0"

        debug_filestore = "0/0"

        debug_journal = "0/0"

        debug_ms = "0/0"

        debug_mon = "0/0"

        debug_monc = "0/0"

        debug_paxos = "0/0"

        debug_tp = "0/0"

        debug_auth = "0/0"

        debug_finisher = "0/0"

        debug_heartbeatmap = "0/0"

        debug_perfcounter = "0/0"

        debug_rgw = "0/0"

        debug_hadoop = "0/0"

        debug_asok = "0/0"

        debug_throttle = "0/0"

 

[osd.0]

        host = node3

        osd data = "" class="">

        bluestore block path = /dev/disk/by-partlabel/osd-device-0-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal

 

[osd.1]

        host = node3

        osd data = "" class="">

        bluestore block path = /dev/disk/by-partlabel/osd-device-1-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-1-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-1-wal

[osd.2]

        host = node3

        osd data = "" class="">

        bluestore block path = /dev/disk/by-partlabel/osd-device-2-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-2-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-2-wal

 

 

[osd.3]

        host = node3

        osd data = "" class="">

        bluestore block path = /dev/disk/by-partlabel/osd-device-3-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-3-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-3-wal

Email Disclaimer & Confidentiality Notice

This message is confidential and intended solely for the use of the recipient to whom they are addressed. If you are not the intended recipient you should not deliver, distribute or copy this e-mail. Please notify the sender immediately by e-mail and delete this e-mail from your system. Copyright © 2016 by Istuary Innovation Labs, Inc. All rights reserved. 

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com