Em qui., 10 de ago. de 2023 às 13:01, Marc <Marc@xxxxxxxxxxxxxxxxx> escreveu: > > I have the following scenario: > > Pool RBD replication x3 > > 5 hosts with 12 SAS spinning disks each > > > > I'm using exactly the following line with FIO to test: > > fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4M -size=10G > > -iodepth=16 -rw=write -filename=./test.img > > > > If I increase the blocksize I can easily reach 1.5 GBps or more. > > > > But when I use blocksize in 4K I get a measly 12 Megabytes per second, > > which is quite annoying. I achieve the same rate if rw=read. > > > > If I use librbd's cache I get a considerable improvement in writing, but > > reading remains the same. > > > > I already tested with rbd_read_from_replica_policy=balance but I didn't > > notice any difference. I tried to leave readahead enabled by setting > > rbd_readahead_disable_after_bytes=0 but I didn't see any difference in > > sequential reading either. > > > > Note: I tested it on another smaller cluster, with 36 SAS disks and got > the > > same result. > > > > I don't know exactly what to look for or configure to have any > improvement. > > What are you expecting? > I expected something a little better (at least in reading), since the other one, with less disks, is showing the same rates. :( > > This is what I have on a vm with an rbd from a hdd pool > > > <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> > I'm using exactly this in libvirt. > > [@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k > -size=1G -iodepth=16 -rw=write -filename=./test.img > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=16 > fio-3.7 > Starting 1 process > Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=57.5MiB/s][r=0,w=14.7k IOPS][eta > 00m:00s] > With writeback I get constant 100 Megs, which is pretty good. I can live with writeback. > > > [@~]# fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k > -size=1G -iodepth=1 -rw=write -filename=./test.img > test: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) > 4096B-4096B, ioengine=libaio, iodepth=1 > fio-3.7 > Starting 1 process > Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=19.9MiB/s][r=0,w=5090 IOPS][eta > 00m:00s] > > > Thanks for showing your results, it's something I can compare to. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |