答复: BlueStore write amplification

[Zhiyuan Wang <zhiyuan.wang@xxxxxxxxxxx>]

Hi

Only one node, and only one nvme SSD, the SSD has 12 partitions, every three for one OSD

And fio is 4k randwrite, iodepth is 128

No snapshot

 

Thanks

 

发件人: Jan Schermer [mailto:jan@xxxxxxxxxxx]
发送时间: 2016年8月23日 14:52
收件人: Zhiyuan Wang <zhiyuan.wang@xxxxxxxxxxx>
抄送: ceph-users@xxxxxxxxxxxxxx
主题: Re: [ceph-users] BlueStore write amplification

 

Is that 400MB on all nodes or on each node? If it's on all nodes then 10:1 is not that surprising.

What what the block size in your fio benchmark?

We had much higher amplification on our cluster with snapshots and stuff...

 

Jan

 

On 23 Aug 2016, at 08:38, Zhiyuan Wang <zhiyuan.wang@xxxxxxxxxxx> wrote:

 

Hi 

I have test bluestore on SSD, and I found that the BW from fio is about 40MB, but

the write BW from iostat of SSD is about 400MB, nearly ten times.

Could someone help to explain this? 

Thanks a lot.

 

Below are my configuration file:

[global]

        fsid = 31e77e3c-447c-4745-a91a-58bda80a868c

        enable experimental unrecoverable data corrupting features = bluestore rocksdb

        osd objectstore = bluestore

 

        bluestore default buffered read = true

        bluestore_min_alloc_size=4096

        osd pool default size = 1

 

        osd pg bits = 8

        osd pgp bits = 8

        auth supported = none

        log to syslog = false

        filestore xattr use omap = true

        auth cluster required = none

        auth service required = none

        auth client required = none

 

        public network = 192.168.200.233/24

        cluster network = 192.168.100.233/24

 

        mon initial members = node3

        mon host = 192.168.200.233

        mon data = "">

          

        filestore merge threshold = 40

        filestore split multiple = 8

        osd op threads = 8

             

        debug_bluefs = "0/0"

        debug_bluestore = "0/0"

        debug_bdev = "0/0" 

        debug_lockdep = "0/0" 

        debug_context = "0/0"  

        debug_crush = "0/0"

        debug_mds = "0/0"

        debug_mds_balancer = "0/0"

        debug_mds_locker = "0/0"

        debug_mds_log = "0/0"

        debug_mds_log_expire = "0/0"

        debug_mds_migrator = "0/0"

        debug_buffer = "0/0"

        debug_timer = "0/0"

        debug_filer = "0/0"

        debug_objecter = "0/0"

        debug_rados = "0/0"

        debug_rbd = "0/0"

        debug_journaler = "0/0"

        debug_objectcacher = "0/0"

        debug_client = "0/0"

        debug_osd = "0/0"

        debug_optracker = "0/0"

        debug_objclass = "0/0"

        debug_filestore = "0/0"

        debug_journal = "0/0"

        debug_ms = "0/0"

        debug_mon = "0/0"

        debug_monc = "0/0"

        debug_paxos = "0/0"

        debug_tp = "0/0"

        debug_auth = "0/0"

        debug_finisher = "0/0"

        debug_heartbeatmap = "0/0"

        debug_perfcounter = "0/0"

        debug_rgw = "0/0"

        debug_hadoop = "0/0"

        debug_asok = "0/0"

        debug_throttle = "0/0"

 

[osd.0]

        host = node3

        osd data = "">

        bluestore block path = /dev/disk/by-partlabel/osd-device-0-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-0-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-wal

 

[osd.1]

        host = node3

        osd data = "">

        bluestore block path = /dev/disk/by-partlabel/osd-device-1-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-1-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-1-wal

[osd.2]

        host = node3

        osd data = "">

        bluestore block path = /dev/disk/by-partlabel/osd-device-2-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-2-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-2-wal

 

 

[osd.3]

        host = node3

        osd data = "">

        bluestore block path = /dev/disk/by-partlabel/osd-device-3-block

        bluestore block db path = /dev/disk/by-partlabel/osd-device-3-db

        bluestore block wal path = /dev/disk/by-partlabel/osd-device-3-wal

Email Disclaimer & Confidentiality Notice

This message is confidential and intended solely for the use of the recipient to whom they are addressed. If you are not the intended recipient you should not deliver, distribute or copy this e-mail. Please notify the sender immediately by e-mail and delete this e-mail from your system. Copyright © 2016 by Istuary Innovation Labs, Inc. All rights reserved. 

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

Email Disclaimer & Confidentiality Notice

This message is confidential and intended solely for the use of the recipient to whom they are addressed. If you are not the intended recipient you should not deliver, distribute or copy this e-mail. Please notify the sender immediately by e-mail and delete this e-mail from your system. Copyright © 2016 by Istuary Innovation Labs, Inc. All rights reserved. 

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com