Hello folks,
we are running a ceph cluster on Luminous consisting of 21 OSD Nodes with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL and DB (1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The cluster is running stable for over a year. We didn’t had a closer look on IO until one of our customers started to complain about a VM we migrated from VMware with Netapp Storage to our Openstack Cloud with ceph storage. He sent us a sysbench report from the machine, which I could reproduce on other VMs as well as on a mounted RBD on physical hardware:
sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G --file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Extra file open flags: 0
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
File operations:
reads/s: 36.36
writes/s: 18.18
fsyncs/s: 2318.59
Throughput:
read, MiB/s: 0.57
written, MiB/s: 0.28
General statistics:
total time: 10.0071s
total number of events: 23755
Latency (ms):
min: 0.01
avg: 6.74
max: 1112.58
95th percentile: 26.68
sum: 160022.67
Threads fairness:
events (avg/stddev): 1484.6875/52.59
execution time (avg/stddev): 10.0014/0.00
Are these numbers reasonable for a cluster of our size?
Best regards
Felix
IT-Services
Telefon 02461 61-9243
E-Mail: f.stolte@xxxxxxxxxxxxx
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
