Hi,
Any idea why 2 servers with one OSD each will provide better performance than 3 ?
Servers are identical
Performance is impacted irrespective if I used SSD for WAL/DB or not
Basically, I am getting lots of cur MB/s zero
Network is separate 10 GB for public and private
I tested it with iperf and I am getting 9.3 Gbs
I have tried replication by 2 and 3 with same results ( much better for 2 servers than 3 )
reinstalled CEPH multiple times
ceph.conf very simple - no major customization ( see below)
I am out of ideas - any hint will be TRULY appreciated
Steven
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 10.10.30.0/24
cluster_network = 192.168.0.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1 # Allow writing 1 copy in a degraded state
osd_crush_chooseleaf_type = 1
[mon]
mon_allow_pool_delete = true
mon_osd_min_down_reporters = 1
[osd]
osd_mkfs_type = xfs
osd_mount_options_xfs = "rw,noatime,nodiratime,attr2,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=4M"
osd_mkfs_options_xfs = "-f -i size=2048"
bluestore_block_db_size = 32212254720
bluestore_block_wal_size = 1073741824
rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects
Object prefix: benchmark_data_osd01_383626
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 57 41 163.991 164 0.197929 0.065543
2 16 57 41 81.992 0 - 0.065543
3 16 67 51 67.9936 20 0.0164632 0.249939
4 16 67 51 50.9951 0 - 0.249939
5 16 71 55 43.9958 8 0.0171439 0.319973
6 16 181 165 109.989 440 0.0159057 0.563746
7 16 182 166 94.8476 4 0.221421 0.561684
8 16 182 166 82.9917 0 - 0.561684
9 16 240 224 99.5458 116 0.0232989 0.638292
10 16 264 248 99.1901 96 0.0222669 0.583336
11 16 264 248 90.1729 0 - 0.583336
12 16 285 269 89.6579 42 0.0165706 0.600606
13 16 285 269 82.7611 0 - 0.600606
14 16 310 294 83.9918 50 0.0254241 0.756351
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
