Re: Slow write speed on 3-node cluster with 6* SATA Harddisks (~ 3.5 MB/s)

[Martin Verges <martin.verges@xxxxxxxx>]

Hello,

harddisks are awful slow. That's normal and expected and result of random io as it would be in a Ceph cluster.

You can speed up raw bandwidth performance using EC but not on such small clusters and not when having high io load.

As you mentioned Proxmox, when it comes to VM workloads spinning media is never an option, use Flash!

--
Martin Verges
Managing director

Mobile: +49 174 9335695
E-Mail: martin.verges@xxxxxxxx
Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263

Web: https://croit.io
YouTube: https://goo.gl/PGE1Bx

Am Mo., 4. Nov. 2019 um 23:44 Uhr schrieb Hermann Himmelbauer <hermann@xxxxxxx>:

Hi,
I recently upgraded my 3-node cluster to proxmox 6 / debian-10 and
recreated my ceph cluster with a new release (14.2.4 bluestore) -
basically hoping to gain some I/O speed.

The installation went flawlessly, reading is faster than before (~ 80
MB/s), however, the write speed is still really slow (~ 3,5 MB/s).

I wonder if I can do anything to speed things up?

My Hardware is as the following:

3 Nodes with Supermicro X8DTT-HIBQF Mainboard each,
2 OSD per node (2TB SATA harddisks, WDC WD2000F9YZ-0),
interconnected via Infiniband 40

The network should be reasonably fast, I measure ~ 16 GBit/s with iperf,
so this seems fine.

I use ceph for RBD only, so my measurement is simply doing a very simple
"dd" read and write test within a virtual machine (Debian 8) like the
following:

read:
dd if=/dev/vdb | pv | dd of=/dev/null
-> 80 MB/s


write:
dd if=/dev/zero | pv | dd of=/dev/vdb
-> 3.5 MB/s

When I do the same on the virtual machine on a disk that is on a NFS
storage, I get something about 30 MB/s for reading and writing.

If I disable the write cache on all OSD disks via "hdparm -W 0
/dev/sdX", I gain a little bit of performance, write speed is then 4.3 MB/s.

Thanks to your help from the list I plan to install a second ceph
cluster which is SSD based (Samsung PM1725b) which should be much
faster, however, I still wonder if there is any way to speed up my
harddisk based cluster?

Thank you in advance for any help,

Best Regards,
Hermann


--
hermann@xxxxxxx
PGP/GPG: 299893C7 (on keyservers)
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx