I am also thinking of moving the wal/db to ssd of the sata hdd's. Did you do tests before and after this change, and know what the difference is iops? And is the advantage more or less when your sata hdd's are slower? -----Original Message----- From: Stolte, Felix [mailto:f.stolte@xxxxxxxxxxxxx] Sent: donderdag 6 juni 2019 10:47 To: ceph-users Subject: Expected IO in luminous Ceph Cluster Hello folks, we are running a ceph cluster on Luminous consisting of 21 OSD Nodes with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL and DB (1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The cluster is running stable for over a year. We didn’t had a closer look on IO until one of our customers started to complain about a VM we migrated from VMware with Netapp Storage to our Openstack Cloud with ceph storage. He sent us a sysbench report from the machine, which I could reproduce on other VMs as well as on a mounted RBD on physical hardware: sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G --file-test-mode=rndrw --file-rw-ratio=2 run sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3) Running the test with following options: Number of threads: 16 Initializing random number generator from current time Extra file open flags: 0 128 files, 8MiB each 1GiB total file size Block size 16KiB Number of IO requests: 0 Read/Write ratio for combined random IO test: 2.00 Periodic FSYNC enabled, calling fsync() each 1 requests. Calling fsync() at the end of test, Enabled. Using synchronous I/O mode Doing random r/w test File operations: reads/s: 36.36 writes/s: 18.18 fsyncs/s: 2318.59 Throughput: read, MiB/s: 0.57 written, MiB/s: 0.28 General statistics: total time: 10.0071s total number of events: 23755 Latency (ms): min: 0.01 avg: 6.74 max: 1112.58 95th percentile: 26.68 sum: 160022.67 Threads fairness: events (avg/stddev): 1484.6875/52.59 execution time (avg/stddev): 10.0014/0.00 Are these numbers reasonable for a cluster of our size? Best regards Felix IT-Services Telefon 02461 61-9243 E-Mail: f.stolte@xxxxxxxxxxxxx ------------------------------------------------------------------------ ------------- ------------------------------------------------------------------------ ------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------ ------------- ------------------------------------------------------------------------ ------------- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |