Hi Patrick,
Some thoughts about blk-mq:
(virtio-blk)
- it's activated by default on kernels >= 3.13 on driver virtio-blk
- The blk-mq feature is currently implemented, and enabled by default, in the following drivers: virtio-blk, mtip32xx, nvme, and rbd. (https://access.redhat.com/doc
umentation/en-us/red_hat_enter )prise_linux/7/html/7.2_release _notes/storage - can be checked with "cat /sys/block/vda/queue/scheduler
", it appears as none - https://serverfault.com/questi
ons/693348/what-does-it-mean- when-linux-has-no-i-o-schedule r
hosts local disk (scsi-mq)
- with disks "sda" (scsi), either rotational or ssd IT'S NOT activated by default on ubuntu (cat /sys/block/sda/queue/scheduler
) - canonical deactivated in >= 3.18 https://bugs.launchpad.
net/ubuntu/+source/linux/+bug/ 1397061 - suse says it does not suit, for scsi rotational, but it's ok for SSD: https://doc.opensuse.org/
documentation/leap/tuning/html /book.sle.tuning/cha.tuning. io.html#cha.tuning.io.scsimq - redhat says: "The scsi-mq feature is provided as a Technology Preview in Red Hat Enterprise Linux 7.2. To enable scsi-mq, specify scsi_mod.use_blk_mq=y on the kernel command line. The default value is n (disabled)." (https://access.redhat.com/doc
umentation/en-us/red_hat_enter )prise_linux/7/html/7.2_release _notes/storage - how to change it: vi /etc/default/grub: GRUB_CMDLINE_LINUX="scsi_mod.u
se_blk_mq=1"; update-grub; reboot
ceph (rbd)
- it's activated by default: The blk-mq feature is currently implemented, and enabled by default, in the following drivers: virtio-blk, mtip32xx, nvme, and rbd. (https://access.redhat.com/doc
umentation/en-us/red_hat_enter )prise_linux/7/html/7.2_release _notes/storage
multipath (device mapper; dm / dm-mpath)
- how to change it: dm_mod.use_blk_mq=y
- deactivated by default, how to verify: To determine whether DM multipath is using blk-mq on a system, cat the file /sys/block/dm-X/dm/use_blk_mq, where dm-X is replaced by the DM multipath device of interest. This file is read-only and reflects what the global value in /sys/module/dm_mod/parameters/
use_blk_mq was at the time the request-based DM multipath device was created . (https://access.redhat.com/documentation/en-us/red_hat_enter )prise_linux/7/html/7.2_release _notes/storage - I thought it would not make any sense, since iscsi is by definition (network) much slower than SSD/NVMe, which is what blk-mq was generated for, but...: It may be beneficial to set dm_mod.use_blk_mq=y if the underlying SCSI devices are also using blk-mq, as doing so reduces locking overhead at the DM layer. (redhat)
observations
- WARNING low performance https://www.
redhat.com/archives/dm-devel/ 2016-February/msg00036.html - request-based device mapper targets planned for 4.1
- now with >= 4.12 linux come with BFQ, a scheduler based en blk-mq
We try in our environment several schedulers but we did't notice a real important improvement, in order to justify a global change in the whole environment. But the best thing is to change/test/document an repeat again and again :)
Hope it helps
Best,
German
2017-12-11 18:17 GMT-03:00 Patrick Fruh <pf@xxxxxxx>:
Hi,
after reading a lot about I/O schedulers and performance gains with blk-mq, I switched to a custom 4.14.5 kernel with CONFIG_SCSI_MQ_DEFAULT enabled to have blk-mq for all devices on my cluster.
This allows me to use the following schedulers for HDDs and SSDs:
mq-deadline, kyber, bfq, none
I’ve currently set the HDD scheduler to bfq and the SSD scheduler to none, however I’m still not sure if this is the best solution performance-wise.
Does anyone have more experience with this and can maybe give me a recommendation? I’m not even sure if blk-mq is a good idea for ceph, since I haven’t really found anything on the topic.
Best,
Patrick
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
