Hi Ilya, thank you for your fast response! Those mkfs parameters I knew, but the possibility to exclude discard from rbd QoS was new to me. It looks like this option is not available with pacific, but with quincy. So we have to upgrade our clusters first. Is it possible to exclude discard by default for ALL rbd images (or all images in a pool) or is it a "per image" setting? If it is a "per rbd image" setting, we will have to extend cinder (openstack) to support it. All the best, Florian ________________________________________ From: Ilya Dryomov <idryomov@xxxxxxxxx> Sent: Wednesday, July 19, 2023 3:16:20 PM To: Engelmann Florian Cc: ceph-users@xxxxxxx Subject: Re: [ceph-users] RBD image QoS rbd_qos_write_bps_limit and rbd_qos_bps_limit and mkfs performance On Wed, Jul 19, 2023 at 11:01 AM Engelmann Florian <florian.engelmann@xxxxxxxxxxxx> wrote: > > Hi, > > I noticed an incredible high performance drop with mkfs.ext4 (as well as mkfs.xfs) when setting (almost) "any" value for rbd_qos_write_bps_limit (or rbd_qos_bps_limit). > > Baseline: 4TB rbd volume rbd_qos_write_bps_limit = 0 > mkfs.ext4: > real 0m6.688s > user 0m0.000s > sys 0m0.006s > > 50GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 53687091200 > mkfs.ext4: > real 1m22.217s > user 0m0.009s > sys 0m0.000s > > 5GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 5368709120 > mkfs.ext4: > real 13m39.770s > user 0m0.008s > sys 0m0.034s > > 500MB/s: 4TB rbd volume rbd_qos_write_bps_limit = 524288000 > mkfs.ext4: > test still runing... I can provide the result if needed. > > The tests are running on a client vm (Ubuntu 22.04) using Qemu/libvirt. > > Using the same values with Qemu/libvirt QoS does not affect mkfs performance. > https://libvirt.org/formatdomain.html#block-i-o-tuning > > Ceph Version: 16.2.11 > Qemu: 6.2.0 > Libvirt: 8.0.0 > Kernel (hypervisor host): 5.19.0-35-generic > librbd1 (hypervisor host): 17.2.5 > > Could anyone pls confirm and explain what's going on? Hi Florian, RBD QoS write limits apply to all write-like operations, including discards. By default, both mkfs.ext4 and mkfs.xfs attempt to discard the entire partition/device and librbd QoS machinery treats that as 4TB worth of writes. RBD images are thin-provisioned, so if you are creating a filesystem on a freshly created image, you can skip discarding with "-E nodiscard" for mkfs.ext4 or "-K" for mkfs.xfs. Alternatively, you can waive QoS limits for discards (or even an arbitrary combination of operations) by setting rbd_qos_exclude_ops option [1] appropriately. [1] https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#confval-rbd_qos_exclude_ops Thanks, Ilya
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
