Hi Sinan,
thanks for the numbers. I am a little bit surprised that your SSD pool has nearly the same stats as you SAS pool.
Nevertheless I would expect our pools to perform like your SAS pool, at least regarding to writes since all our write ops should be placed on our SSDs. But since I only achieve 10% of your numbers I need to figure out my bottle neck. For now I have no clue. According to our monitoring system network bandwith, ram or cpu usage is even close to be saturated.
Could someone advice me on where to look?
Regards Felix
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
-------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------
Am 07.06.19, 13:33 schrieb "Sinan Polat" <sinan@xxxxxxxx>:
Hi Felix,
I have 2 Pools, a SSD only and a SAS only pool.
SSD pool is spread over 12 OSD servers.
SAS pool is spread over 6 OSD servers.
See results (SSD Only Pool):
# sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
--file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.0.17 (using system LuaJIT 2.0.4)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...
Threads started!
File operations:
reads/s: 508.38
writes/s: 254.19
fsyncs/s: 32735.14
Throughput:
read, MiB/s: 7.94
written, MiB/s: 3.97
General statistics:
total time: 10.0103s
total number of events: 333336
Latency (ms):
min: 0.00
avg: 0.48
max: 10.18
95th percentile: 2.11
sum: 159830.07
Threads fairness:
events (avg/stddev): 20833.5000/335.70
execution time (avg/stddev): 9.9894/0.00
#
See results (SAS Only Pool):
# sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
--file-test-mode=rndrw --file-rw-ratio=2 run
sysbench 1.0.17 (using system LuaJIT 2.0.4)
Running the test with following options:
Number of threads: 16
Initializing random number generator from current time
Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 2.00
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...
Threads started!
File operations:
reads/s: 490.11
writes/s: 245.10
fsyncs/s: 31565.00
Throughput:
read, MiB/s: 7.66
written, MiB/s: 3.83
General statistics:
total time: 10.0143s
total number of events: 321477
Latency (ms):
min: 0.00
avg: 0.50
max: 20.50
95th percentile: 2.30
sum: 159830.82
Threads fairness:
events (avg/stddev): 20092.3125/186.66
execution time (avg/stddev): 9.9894/0.00
#
Kind regards,
Sinan Polat
> Op 7 juni 2019 om 12:47 schreef "Stolte, Felix" <f.stolte@xxxxxxxxxxxxx>:
>
>
> Hi Sinan,
>
> that would be great. The numbers should differ a lot, since you have an all
> flash pool, but it would be interesting, what we could expect from such a
> configuration.
>
> Regards
> Felix
>
> -------------------------------------------------------------------------------------
> -------------------------------------------------------------------------------------
> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Prof. Dr. Sebastian M. Schmidt
> -------------------------------------------------------------------------------------
> -------------------------------------------------------------------------------------
>
>
> Am 07.06.19, 12:02 schrieb "Sinan Polat" <sinan@xxxxxxxx>:
>
> Hi Felix,
>
> I can run your commands inside an OpenStack VM. Tthe storage cluster
> contains of 12 OSD servers, holding each 8x 960GB SSD. Luminous FileStore.
> Replicated 3.
>
> Would it help you to run your command on my cluster?
>
> Sinan
>
> > Op 7 jun. 2019 om 08:52 heeft Stolte, Felix <f.stolte@xxxxxxxxxxxxx> het
> > volgende geschreven:
> >
> > I have no performance data before we migrated to bluestore. You should
> > start a separate topic regarding your question.
> >
> > Could anyone with an more or less equally sized cluster post the output
> > of a sysbench with the following parameters (either from inside an
> > openstack vm or a mounted rbd)?
> >
> > sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
> > --file-test-mode=rndrw --file-rw-ratio=2 prepare
> >
> > sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
> > --file-test-mode=rndrw --file-rw-ratio=2 run
> >
> > Thanks in advance.
> >
> > Regards
> > Felix
> >
> > -------------------------------------------------------------------------------------
> > -------------------------------------------------------------------------------------
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof. Dr. Sebastian M. Schmidt
> > -------------------------------------------------------------------------------------
> > -------------------------------------------------------------------------------------
> >
> >
> > Am 06.06.19, 15:09 schrieb "Marc Roos" <M.Roos@xxxxxxxxxxxxxxxxx>:
> >
> >
> > I am also thinking of moving the wal/db to ssd of the sata hdd's. Did
> >
> > you do tests before and after this change, and know what the
> > difference
> > is iops? And is the advantage more or less when your sata hdd's are
> > slower?
> >
> >
> > -----Original Message-----
> > From: Stolte, Felix [mailto:f.stolte@xxxxxxxxxxxxx]
> > Sent: donderdag 6 juni 2019 10:47
> > To: ceph-users
> > Subject: Expected IO in luminous Ceph Cluster
> >
> > Hello folks,
> >
> > we are running a ceph cluster on Luminous consisting of 21 OSD Nodes
> > with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL and DB
> >
> > (1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The
> > cluster
> > is running stable for over a year. We didn’t had a closer look on IO
> > until one of our customers started to complain about a VM we migrated
> >
> > from VMware with Netapp Storage to our Openstack Cloud with ceph
> > storage. He sent us a sysbench report from the machine, which I could
> >
> > reproduce on other VMs as well as on a mounted RBD on physical
> > hardware:
> >
> > sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G
> >
> > --file-test-mode=rndrw --file-rw-ratio=2 run sysbench 1.0.11 (using
> > system LuaJIT 2.1.0-beta3)
> >
> > Running the test with following options:
> > Number of threads: 16
> > Initializing random number generator from current time
> >
> > Extra file open flags: 0
> > 128 files, 8MiB each
> > 1GiB total file size
> > Block size 16KiB
> > Number of IO requests: 0
> > Read/Write ratio for combined random IO test: 2.00 Periodic FSYNC
> > enabled, calling fsync() each 1 requests.
> > Calling fsync() at the end of test, Enabled.
> > Using synchronous I/O mode
> > Doing random r/w test
> >
> > File operations:
> > reads/s: 36.36
> > writes/s: 18.18
> > fsyncs/s: 2318.59
> >
> > Throughput:
> > read, MiB/s: 0.57
> > written, MiB/s: 0.28
> >
> > General statistics:
> > total time: 10.0071s
> > total number of events: 23755
> >
> > Latency (ms):
> > min: 0.01
> > avg: 6.74
> > max: 1112.58
> > 95th percentile: 26.68
> > sum: 160022.67
> >
> > Threads fairness:
> > events (avg/stddev): 1484.6875/52.59
> > execution time (avg/stddev): 10.0014/0.00
> >
> > Are these numbers reasonable for a cluster of our size?
> >
> > Best regards
> > Felix
> > IT-Services
> > Telefon 02461 61-9243
> > E-Mail: f.stolte@xxxxxxxxxxxxx
> >
> > ------------------------------------------------------------------------
> > -------------
> >
> > ------------------------------------------------------------------------
> > -------------
> > Forschungszentrum Juelich GmbH
> > 52425 Juelich
> > Sitz der Gesellschaft: Juelich
> > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
> > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >
> > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> > Prof.
> > Dr. Sebastian M. Schmidt
> >
> > ------------------------------------------------------------------------
> > -------------
> >
> > ------------------------------------------------------------------------
> > -------------
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
