Hi Florian, On 3/7/19 10:27 AM, Florian Engelmann wrote: > > So the settings are recognized and used by qemu. But any value higher > than the default (32MB) of the cache size leads to strange IOPS > results. IOPS are very constant with 32MB ~20.000 - 23.000 but if we > define a bigger cache size (we tested from 64MB up to 256MB) the IOPS > get very unconstant (from 0 IOPS up to 23.000). > > Setting "rbd cache max dirty" to 0 changes the behaviour to > write-through as far as I understood. I expected the latency to > increase to at least 0.6 ms what was the case but I also expected the > IOPS to increase to up to 60.000 which was not the case. IOPS was > constant at ~ 14.000IOPS (4 jobs, QD=64). > > > > Am 3/7/19 um 11:41 AM schrieb Florian Engelmann: >> Hi, >> >> we are running an Openstack environment with Ceph block storage. >> There are six nodes in the current Ceph cluster (12.2.10) with NVMe >> SSDs and a P4800X Optane for rocksdb and WAL. >> The decision was made to use rbd writeback cache with KVM/QEMU. The >> write latency is incredible good (~85 µs) and the read latency is >> still good (~0.6ms). But we are limited to ~23.000 IOPS in a KVM >> machine. So we did the same FIO benchmark after we disabled the rbd >> cache and got 65.000 IOPS but of course the write latency (QD1) was >> increased to ~ 0.6ms. How does fio with rbd cache enabled compare to QEMU/KVM? Can you try making sure the queue depth used with QEMU is similar to what you're using with fio? You'll probably have to play with the configs of the disks in libvirt, there are a couple of settings you can try, I'm unfortunately not very familiar with them (num-queues, iothreads [1], and maybe others). I think I'd start there. Mohamad [1] https://libvirt.org/formatdomain.html#elementsIOThreadsAllocation _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |