Hello, We are in the process of evaluating the performance of a testing cluster (3 nodes) with ceph jewel. Our setup consists of: 3 monitors (VMs) 2 physical servers each connected with 1 JBOD running Ubuntu Server 16.04 Each server has 32 threads @2.1GHz and 128GB RAM. The disk distribution per server is: 38 * HUS726020ALS210 (SAS rotational) 2 * HUSMH8010BSS200 (SAS SSD for journals) 2 * ST1920FM0043 (SAS SSD for data) 1 * INTEL SSDPEDME012T4 (NVME measured with fio ~300K iops) Since we don't currently have a 10Gbit switch, we test the performance with the cluster in a degraded state, the noout flag set and we mount rbd images on the powered on osd node. We confirmed that the network is not saturated during the tests. We ran tests on the NVME disk and the pool created on this disk where we hoped to get the most performance without getting limited by the hardware specs since we have more disks than CPU threads. The nvme disk was at first partitioned with one partition and the journal on the same disk. The performance on random 4K reads was topped at 50K iops. We then removed the osd and partitioned with 4 data partitions and 4 journals on the same disk. The performance didn't increase significantly. Also, since we run read tests, the journals shouldn't cause performance issues. We then ran 4 fio processes in parallel on the same rbd mounted image and the total iops reached 100K. More parallel fio processes didn't increase the measured iops. Our ceph.conf is pretty basic (debug is set to 0/0 for everything) and the crushmap just defines the different buckets/rules for the disk separation (rotational, ssd, nvme) in order to create the required pools Is the performance of 100.000 iops for random 4K read normal for a disk that on the same benchmark runs at more than 300K iops on the same hardware or are we missing something? Best regards, Kostas _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |