Thanks for helping
I thought that with CEPH, the more servers you have the better the performance
- that is why I am so confused
Also, I tried to add the 4th server ( still no luck - in fact the rado bench output I included was from 4 servers, one OSD on each, bluestore, replication 2 )
Here is the same rados bench but with only 2 servers/ 2 OSDs
rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects
Object prefix: benchmark_data_osd01_384454
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 101 85 339.981 340 0.244679 0.160362
2 16 149 133 265.972 192 0.303997 0.226268
3 16 197 181 241.307 192 0.179013 0.255609
4 16 241 225 224.975 176 0.353464 0.272623
5 16 289 273 218.376 192 0.303821 0.282425
6 16 338 322 214.643 196 0.326009 0.29105
7 16 387 371 211.977 196 0.27048 0.296497
8 16 436 420 209.977 196 0.287188 0.299224
9 16 479 463 205.755 172 0.380512 0.302272
10 16 527 511 204.378 192 0.289532 0.306163
11 16 576 560 203.614 196 0.406783 0.309271
12 16 624 608 202.645 192 0.282266 0.312167
13 16 667 651 200.286 172 0.377555 0.313733
14 16 716 700 199.978 196 0.350938 0.315445
15 16 764 748 199.445 192 0.183931 0.317474
Here is my iperf3 results
0.00-1.00 sec 1.15 GBytes 9.92 Gbits/sec 0 850 KBytes
[ 4] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 850 KBytes
On Thu, 19 Apr 2018 at 08:28, Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx> wrote:
If I may guess, because with 3 it reads from 3 and with 2 it reads only
from 2. You should be able to verify this with something like dstat -d
-D sda,sdb,sdc,sdd,sde,sdf,sdg not?
With replication of 2, objects are still being stored among the 3 nodes.
I am getting with iperf3 on 10Gbit
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.31 MBytes
[ 4] 10.00-20.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.79 MBytes
-----Original Message-----
From: Steven Vacaroaia [mailto:stef97@xxxxxxxxx]
Sent: donderdag 19 april 2018 14:11
To: ceph-users
Subject: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?
Hi,
Any idea why 2 servers with one OSD each will provide better performance
than 3 ?
Servers are identical
Performance is impacted irrespective if I used SSD for WAL/DB or not
Basically, I am getting lots of cur MB/s zero
Network is separate 10 GB for public and private I tested it with iperf
and I am getting 9.3 Gbs
I have tried replication by 2 and 3 with same results ( much better for
2 servers than 3 )
reinstalled CEPH multiple times
ceph.conf very simple - no major customization ( see below) I am out of
ideas - any hint will be TRULY appreciated
Steven
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 10.10.30.0/24
cluster_network = 192.168.0.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1 # Allow writing 1 copy in a degraded state
osd_crush_chooseleaf_type = 1
[mon]
mon_allow_pool_delete = true
mon_osd_min_down_reporters = 1
[osd]
osd_mkfs_type = xfs
osd_mount_options_xfs =
"rw,noatime,nodiratime,attr2,logbufs=8,logbsize=256k,largeio,inode64,swa
lloc,allocsize=4M"
osd_mkfs_options_xfs = "-f -i size=2048"
bluestore_block_db_size = 32212254720
bluestore_block_wal_size = 1073741824
rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq
hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects
of size 4194304 for up to 120 seconds or 0 objects Object prefix:
benchmark_data_osd01_383626
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 16 57 41 163.991 164 0.197929
0.065543
2 16 57 41 81.992 0 -
0.065543
3 16 67 51 67.9936 20 0.0164632
0.249939
4 16 67 51 50.9951 0 -
0.249939
5 16 71 55 43.9958 8 0.0171439
0.319973
6 16 181 165 109.989 440 0.0159057
0.563746
7 16 182 166 94.8476 4 0.221421
0.561684
8 16 182 166 82.9917 0 -
0.561684
9 16 240 224 99.5458 116 0.0232989
0.638292
10 16 264 248 99.1901 96 0.0222669
0.583336
11 16 264 248 90.1729 0 -
0.583336
12 16 285 269 89.6579 42 0.0165706
0.600606
13 16 285 269 82.7611 0 -
0.600606
14 16 310 294 83.9918 50 0.0254241
0.756351
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
