Hi, I created 40 osds ceph cluster with 8 PM863 960G SSD as journal. One ssd is used by 5 osd drives as journal. The ssd 512 random write performance is about 450MB/s, but the whole cluster sequential write throughput is only 800MB/s. Any suggestion on improving sequential write performance? thanks.
Testing result is here:
rados bench -p libvirt-pool 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_redpower-sh-04_16462
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 189 174 695.968 696 0.0359122 0.082477
2 16 395 379 757.938 820 0.0634079 0.0826266
3 16 582 566 754.601 748 0.0401129 0.0830207
4 16 796 780 779.934 856 0.0374938 0.0816794
5 16 977 961 768.735 724 0.0489886 0.0827479
6 16 1172 1156 770.601 780 0.0428639 0.0812062
7 16 1387 1371 783.362 860 0.0461826 0.0811803
8 16 1545 1529 764.433 632 0.238497 0.0831018
9 16 1765 1749 777.265 880 0.0557358 0.0814399
10 16 1971 1955 781.931 824 0.0321333 0.0814144
Total time run: 10.044813
Total writes made: 1972
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 785.281
Stddev Bandwidth: 80.8235
Max bandwidth (MB/sec): 880
Min bandwidth (MB/sec): 632
Average IOPS: 196
Stddev IOPS: 20
Max IOPS: 220
Min IOPS: 158
Average Latency(s): 0.081415
Stddev Latency(s): 0.0554568
Max latency(s): 0.345111
Min latency(s): 0.0230153
my ceph osd configuration:Testing result is here:
rados bench -p libvirt-pool 10 write --no-cleanup
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_redpower-sh-04_16462
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 189 174 695.968 696 0.0359122 0.082477
2 16 395 379 757.938 820 0.0634079 0.0826266
3 16 582 566 754.601 748 0.0401129 0.0830207
4 16 796 780 779.934 856 0.0374938 0.0816794
5 16 977 961 768.735 724 0.0489886 0.0827479
6 16 1172 1156 770.601 780 0.0428639 0.0812062
7 16 1387 1371 783.362 860 0.0461826 0.0811803
8 16 1545 1529 764.433 632 0.238497 0.0831018
9 16 1765 1749 777.265 880 0.0557358 0.0814399
10 16 1971 1955 781.931 824 0.0321333 0.0814144
Total time run: 10.044813
Total writes made: 1972
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 785.281
Stddev Bandwidth: 80.8235
Max bandwidth (MB/sec): 880
Min bandwidth (MB/sec): 632
Average IOPS: 196
Stddev IOPS: 20
Max IOPS: 220
Min IOPS: 158
Average Latency(s): 0.081415
Stddev Latency(s): 0.0554568
Max latency(s): 0.345111
Min latency(s): 0.0230153
sd_mkfs_type = xfs
osd_mount_options_xfs = rw,noatime,inode64,logbsize=256k
osd_mkfs_options_xfs = -f -i size=2048
filestore_max_inline_xattr_size = 254
filestore_max_inline_xattrs = 6
osd_op_threads = 20
filestore_queue_max_ops = 25000
journal_max_write_entries=10000
journal_queue_max_ops=50000
objecter_inflight_ops=10240
filestore_queue_max_bytes=1048576000
filestore_queue_committing_max_bytes =1048576000
journal_max_write_bytes=1073714824
journal_queue_max_bytes=10485760000
ms_dispatch_throttle_bytes=1048576000
objecter_infilght_op_bytes=1048576000
filestore_max_sync_interval=20
filestore_flusher=false
filestore_flush_min=0
filestore_sync_flush=true
journal_block_align = true
journal_dio = true
journal_aio = true
journal_force_aio = true
osd_op_num_shards=8
osd_op_num_threads_per_shard=2
filestore_wbthrottle_enable=false
filestore_fd_cache_size=1024
filestore_omap_header_cache_size=1024
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
