Another frequent cause is when someone runs from an RBD VM that is IOPS-throttled at the libvirt layer, and they do like 1KB writes. Increasing the write size does wonders :D > On Dec 10, 2024, at 5:47 AM, Ml Ml <mliebherr99@xxxxxxxxxxxxxx> wrote: > > Thanks a lot! > That was the part i did wrong :) > > Cheers, > Mario > > On Mon, Dec 9, 2024 at 5:38 PM <darren@xxxxxxxxxxxx> wrote: >> >> Hi Mario, >> >> You are not testing like for like. >> >> Either increase the numjobs on the FIO command to 16 or reduce the io-threads on the rbd bench command to 1 or add on iodepth=16 to the FIO command. >> >> Currently with the RBD bench you have 16 concurrent IO’s running at the same time but you are restricting FIO to 1. >> >> Darren >> >>> On 9 Dec 2024, at 15:31, Ml Ml <mliebherr99@xxxxxxxxxxxxxx> wrote: >>> >>> Hi all, >>> >>> I have a question regarding the performance differences between rbd >>> bench and a fio test on a Ceph RBD. >>> During testing of my Ceph environment, I noticed that rbd bench >>> reports significantly higher write speeds compared to fio, >>> which is run on an XFS filesystem mounted on the same RBD. >>> >>> Here are the test details: >>> >>> fio test: >>> ------------ >>> fio --name=ceph_bench --rw=write --bs=4M --direct=1 --ioengine=libaio >>> --size=4G --numjobs=1 --filename=/mnt/ceph10/backup-cluster1/testfile >>> >>> Results: >>> Write speed: ~70.3 MiB/s (73.7 MB/s) >>> IOPS: ~17 >>> Latency: ~56.4 ms (average) >>> >>> rbd bench: >>> -------------- >>> rbd bench --io-type write --io-size 4M --io-threads 16 --io-total 12G >>> backup-proxmox/cluster1-new --keyring >>> /etc/ceph/ceph10/ceph.client.admin.keyring --conf >>> /etc/ceph/ceph10/ceph.conf >>> >>> Results: >>> Write speed: ~150–180 MiB/s (sustained after initial peak) >>> IOPS: ~43–46 >>> Threads: 16 >>> >>> ceph tell osd.X bench: >>> --------------------------- >>> osd.0: IOPS: 20.03, Throughput: 80.11 MB/s >>> osd.1: IOPS: 23.08, Throughput: 92.33 MB/s >>> osd.2: IOPS: 22.01, Throughput: 88.02 MB/s >>> osd.3: IOPS: 17.65, Throughput: 70.61 MB/s >>> osd.4: IOPS: 20.15, Throughput: 80.59 MB/s >>> osd.5: IOPS: 19.71, Throughput: 78.82 MB/s >>> osd.6: IOPS: 18.96, Throughput: 75.84 MB/s >>> osd.7: IOPS: 20.13, Throughput: 80.52 MB/s >>> osd.8: IOPS: 16.45, Throughput: 65.79 MB/s (hm) >>> osd.9: IOPS: 31.16, Throughput: 124.65 MB/s >>> osd.10: IOPS: 23.19, Throughput: 92.76 MB/s >>> osd.12: IOPS: 19.10, Throughput: 76.38 MB/s >>> osd.13: IOPS: 28.02, Throughput: 112.09 MB/s >>> osd.14: IOPS: 19.07, Throughput: 76.27 MB/s >>> osd.15: IOPS: 21.40, Throughput: 85.59 MB/s >>> osd.16: IOPS: 20.24, Throughput: 80.95 MB/s >>> >>> >>> I understand that rbd bench is designed to test the raw performance of >>> the RBD layer, while fio runs at the filesystem level. However, the >>> large discrepancy between the two results has left me curious. What >>> could be causing this difference? Are there any specific factors >>> related to caching, journaling, or XFS itself that could explain the >>> significantly lower performance in the fio test? >>> >>> Any insights or recommendations for further troubleshooting would be >>> greatly appreciated! >>> >>> >>> >>> Thanks in advance! >>> Best regards, >>> Mario >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |