Re: ceph write performance issue

[Burkhard Linke <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I would propose to start with a OSD only benchmark (ceph tell osd.* bench) to get an upper estimate of what the OSD itself is capabable of.

You also did not describe the network setup. 800MB/s is a good value if the network connection is a 10GbE link (which has a theoretical upper limit of 1.2 GB/s without protocol overhead). You may also be limited by the client's CPU, so check CPU load, too.

As a comparison, our cluster (9 hosts with single 40 GbE links) allows up to 1 GB/s in simple rados benchmarks runing on clients with 2x 10GbE links).

Regards,

Burkhard

On 09/29/2016 12:05 PM, min fang wrote:

I used 2 copies, not 3, so should be 1000MB/s in theory. thanks.

2016-09-29 17:54 GMT+08:00 Nick Fisk <nick@xxxxxxxxxx>:

From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of min fang
Sent: 29 September 2016 10:34
To: ceph-users <ceph-users@xxxxxxxxxxxxxx>
Subject: ceph write performance issue

 

Hi, I created 40 osds ceph cluster with 8 PM863 960G SSD as journal. One ssd is used by 5 osd drives as journal.   The ssd 512 random write performance is about 450MB/s, but the whole cluster sequential write throughput is only 800MB/s. Any suggestion on improving sequential write performance? thanks.

 

Take a conservative figure of 50MB/s for each disk as writing in Ceph is not just straight sequential writes, there is a slight random nature to it.

(40x50MB/s)/3 = 666MB/s. Seems fine to me.

 

Testing result is here:
rados bench -p libvirt-pool 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_redpower-sh-04_16462
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15       189       174   695.968       696   0.0359122    0.082477
    2      16       395       379   757.938       820   0.0634079   0.0826266
    3      16       582       566   754.601       748   0.0401129   0.0830207
    4      16       796       780   779.934       856   0.0374938   0.0816794
    5      16       977       961   768.735       724   0.0489886   0.0827479
    6      16      1172      1156   770.601       780   0.0428639   0.0812062
    7      16      1387      1371   783.362       860   0.0461826   0.0811803
    8      16      1545      1529   764.433       632    0.238497   0.0831018
    9      16      1765      1749   777.265       880   0.0557358   0.0814399
   10      16      1971      1955   781.931       824   0.0321333   0.0814144
Total time run:         10.044813
Total writes made:      1972
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     785.281
Stddev Bandwidth:       80.8235
Max bandwidth (MB/sec): 880
Min bandwidth (MB/sec): 632
Average IOPS:           196
Stddev IOPS:            20
Max IOPS:               220
Min IOPS:               158
Average Latency(s):     0.081415
Stddev Latency(s):      0.0554568
Max latency(s):         0.345111
Min latency(s):         0.0230153

my ceph osd configuration:
sd_mkfs_type = xfs
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
osd_mkfs_options_xfs = -f -i size=2048
filestore_max_inline_xattr_size = 254
filestore_max_inline_xattrs = 6
osd_op_threads = 20
filestore_queue_max_ops = 25000
journal_max_write_entries=10000
journal_queue_max_ops=50000
objecter_inflight_ops=10240
filestore_queue_max_bytes=1048576000
filestore_queue_committing_max_bytes =1048576000
journal_max_write_bytes=1073714824
journal_queue_max_bytes=10485760000
ms_dispatch_throttle_bytes=1048576000
objecter_infilght_op_bytes=1048576000
filestore_max_sync_interval=20
filestore_flusher=false
filestore_flush_min=0
filestore_sync_flush=true
journal_block_align = true
journal_dio = true
journal_aio = true
journal_force_aio = true
osd_op_num_shards=8
osd_op_num_threads_per_shard=2
filestore_wbthrottle_enable=false
filestore_fd_cache_size=1024
filestore_omap_header_cache_size=1024

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Dr. rer. nat. Burkhard Linke
Bioinformatics and Systems Biology
Justus-Liebig-University Giessen
35392 Giessen, Germany
Phone: (+49) (0)641 9935810

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com