I've just worked out I had the same issue, been trying to work out the cause for the past few days!
However I am using brand new enterprise Toshiba drivers with 256MB write cache, was seeing I/O wait peaks of 40% even during a small writing operation to CEPH and commit / apply latency's in the 40ms+.
Just went through and disabled the write cache on each drive, and done a few tests with the exact same write performance, but I/O wait in the <1% and commit / apply latency's in the 1-3ms max.
Something somewhere definitely doesn't seem to like the write cache being enabled on the disks, this is a EC Pool in the latest Mimic version.
On Sun, Nov 11, 2018 at 5:34 AM Vitaliy Filippov <vitalif@xxxxxxxxxx> wrote:
Hi
A weird thing happens in my test cluster made from desktop hardware.
The command `for i in /dev/sd?; do hdparm -W 0 $i; done` increases
single-thread write iops (reduces latency) 7 times!
It is a 3-node cluster with Ryzen 2700 CPUs, 3x SATA 7200rpm HDDs + 1x
SATA desktop SSD for system and ceph-mon + 1x SATA server SSD for
block.db/wal in each host. Hosts are linked by 10gbit ethernet (not the
fastest one though, average RTT according to flood-ping is 0.098ms). Ceph
and OpenNebula are installed on the same hosts, OSDs are prepared with
ceph-volume and bluestore with default options. SSDs have capacitors
('power-loss protection'), write cache is turned off for them since the
very beginning (hdparm -W 0 /dev/sdb). They're quite old, but each of them
is capable of delivering ~22000 iops in journal mode (fio -sync=1
-direct=1 -iodepth=1 -bs=4k -rw=write).
However, RBD single-threaded random-write benchmark originally gave awful
results - when testing with `fio -ioengine=libaio -size=10G -sync=1
-direct=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=60
-filename=./testfile` from inside a VM, the result was only 58 iops
average (17ms latency). This was not what I expected from the HDD+SSD
setup.
But today I tried to play with cache settings for data disks. And I was
really surprised to discover that just disabling HDD write cache (hdparm
-W 0 /dev/sdX for all HDD devices) increases single-threaded performance
~7 times! The result from the same VM (without even rebooting it) is
iops=405, avg lat=2.47ms. That's a magnitude faster and in fact 2.5ms
seems sort of an expected number.
As I understand 4k writes are always deferred at the default setting of
prefer_deferred_size_hdd=32768, this means they should only get written to
the journal device before OSD acks the write operation.
So my question is WHY? Why does HDD write cache affect commit latency with
WAL on an SSD?
I would also appreciate if anybody with similar setup (HDD+SSD with
desktop SATA controllers or HBA) could test the same thing...
--
With best regards,
Vitaliy Filippov
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
