Hi all, I'm trying to troubleshoot an interesting problem with RBD performance for VMs. Tests were done using fio both outside and inside the VMs shows that random read/write is 20-30% slower than bulk read/write at QD=1. However, at QD=16/32/64, random read/write is sometimes 3X faster than bulk read/write. Inside the VMs, tests were done with -direct=1 -sync=1 using libaio. Outside VMs, test were done with -direct=1 -sync=1 with both librbd and libaio. The gap between random and bulk I/O narrows with increasing QD to 128. However, there's always a 20-30% difference with random I/O being faster. Read and write tests show similar results both inside and outside VMs. Typically, the random I/O performance would be less (or much less) than bulk. Any idea as to what I should be looking at? Thanks. Tri Hoang Inside VM ======== At QD=8, randread is around 2.8X read @ 64k --------------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=762MiB/s][r=12.2k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1551: Wed Sep 30 08:19:56 2020 read: IOPS=11.9k, BW=743MiB/s (779MB/s)(4096MiB/5515msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=8 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=268MiB/s][r=4289 IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1554: Wed Sep 30 08:22:03 2020 read: IOPS=4374, BW=273MiB/s (287MB/s)(4096MiB/14981msec) At QD=128, randread is around 1.4X read --------------------------------------------------------- tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=randread -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1) read: (groupid=0, jobs=1): err= 0: pid=1548: Wed Sep 30 08:18:59 2020 read: IOPS=23.1k, BW=1441MiB/s (1511MB/s)(4096MiB/2843msec) tri@ansible:~$ fio -name=read -ioengine=libaio -iodepth=128 -direct=1 -sync=1 -rw=read -bs=64k -size=4G -runtime=120 --filename=test.fio read: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.12 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=974MiB/s][r=15.6k IOPS][eta 00m:00s] read: (groupid=0, jobs=1): err= 0: pid=1545: Wed Sep 30 08:17:38 2020 read: IOPS=15.9k, BW=997MiB/s (1045MB/s)(4096MiB/4110msec) _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |