If I may guess, because with 3 it reads from 3 and with 2 it reads only from 2. You should be able to verify this with something like dstat -d -D sda,sdb,sdc,sdd,sde,sdf,sdg not? With replication of 2, objects are still being stored among the 3 nodes. I am getting with iperf3 on 10Gbit [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.31 MBytes [ 4] 10.00-20.00 sec 11.5 GBytes 9.89 Gbits/sec 0 1.79 MBytes -----Original Message----- From: Steven Vacaroaia [mailto:stef97@xxxxxxxxx] Sent: donderdag 19 april 2018 14:11 To: ceph-users Subject: ceph luminous 12.2.4 - 2 servers better than 3 ? Hi, Any idea why 2 servers with one OSD each will provide better performance than 3 ? Servers are identical Performance is impacted irrespective if I used SSD for WAL/DB or not Basically, I am getting lots of cur MB/s zero Network is separate 10 GB for public and private I tested it with iperf and I am getting 9.3 Gbs I have tried replication by 2 and 3 with same results ( much better for 2 servers than 3 ) reinstalled CEPH multiple times ceph.conf very simple - no major customization ( see below) I am out of ideas - any hint will be TRULY appreciated Steven auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 10.10.30.0/24 cluster_network = 192.168.0.0/24 osd_pool_default_size = 2 osd_pool_default_min_size = 1 # Allow writing 1 copy in a degraded state osd_crush_chooseleaf_type = 1 [mon] mon_allow_pool_delete = true mon_osd_min_down_reporters = 1 [osd] osd_mkfs_type = xfs osd_mount_options_xfs = "rw,noatime,nodiratime,attr2,logbufs=8,logbsize=256k,largeio,inode64,swa lloc,allocsize=4M" osd_mkfs_options_xfs = "-f -i size=2048" bluestore_block_db_size = 32212254720 bluestore_block_wal_size = 1073741824 rados bench -p rbd 120 write --no-cleanup && rados bench -p rbd 120 seq hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 120 seconds or 0 objects Object prefix: benchmark_data_osd01_383626 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 57 41 163.991 164 0.197929 0.065543 2 16 57 41 81.992 0 - 0.065543 3 16 67 51 67.9936 20 0.0164632 0.249939 4 16 67 51 50.9951 0 - 0.249939 5 16 71 55 43.9958 8 0.0171439 0.319973 6 16 181 165 109.989 440 0.0159057 0.563746 7 16 182 166 94.8476 4 0.221421 0.561684 8 16 182 166 82.9917 0 - 0.561684 9 16 240 224 99.5458 116 0.0232989 0.638292 10 16 264 248 99.1901 96 0.0222669 0.583336 11 16 264 248 90.1729 0 - 0.583336 12 16 285 269 89.6579 42 0.0165706 0.600606 13 16 285 269 82.7611 0 - 0.600606 14 16 310 294 83.9918 50 0.0254241 0.756351 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |