On Wed, Jul 19, 2023 at 11:01 AM Engelmann Florian <florian.engelmann@xxxxxxxxxxxx> wrote: > > Hi, > > I noticed an incredible high performance drop with mkfs.ext4 (as well as mkfs.xfs) when setting (almost) "any" value for rbd_qos_write_bps_limit (or rbd_qos_bps_limit). > > Baseline: 4TB rbd volume rbd_qos_write_bps_limit = 0 > mkfs.ext4: > real 0m6.688s > user 0m0.000s > sys 0m0.006s > > 50GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 53687091200 > mkfs.ext4: > real 1m22.217s > user 0m0.009s > sys 0m0.000s > > 5GB/s: 4TB rbd volume rbd_qos_write_bps_limit = 5368709120 > mkfs.ext4: > real 13m39.770s > user 0m0.008s > sys 0m0.034s > > 500MB/s: 4TB rbd volume rbd_qos_write_bps_limit = 524288000 > mkfs.ext4: > test still runing... I can provide the result if needed. > > The tests are running on a client vm (Ubuntu 22.04) using Qemu/libvirt. > > Using the same values with Qemu/libvirt QoS does not affect mkfs performance. > https://libvirt.org/formatdomain.html#block-i-o-tuning > > Ceph Version: 16.2.11 > Qemu: 6.2.0 > Libvirt: 8.0.0 > Kernel (hypervisor host): 5.19.0-35-generic > librbd1 (hypervisor host): 17.2.5 > > Could anyone pls confirm and explain what's going on? Hi Florian, RBD QoS write limits apply to all write-like operations, including discards. By default, both mkfs.ext4 and mkfs.xfs attempt to discard the entire partition/device and librbd QoS machinery treats that as 4TB worth of writes. RBD images are thin-provisioned, so if you are creating a filesystem on a freshly created image, you can skip discarding with "-E nodiscard" for mkfs.ext4 or "-K" for mkfs.xfs. Alternatively, you can waive QoS limits for discards (or even an arbitrary combination of operations) by setting rbd_qos_exclude_ops option [1] appropriately. [1] https://docs.ceph.com/en/latest/rbd/rbd-config-ref/#confval-rbd_qos_exclude_ops Thanks, Ilya _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |