Hi, I have done fio tests on multiple qemu setups and host machine, to see if librbd use more cpu than krbd and also find the best qemu optimisations. Fio test was done with blocksize=4K , read and randread with num_jobs=32. 1) First, I have done more tests with qemu optimisation, like iothread (dataplane) for virtio-blk disk, and num_queue=8 for virtio-scsi disks. for virtio iothread: -------------------- It's working only with qemu + krbd, and for sequential reads. And It's seem that qemu aggregate reads in bigger ceph reads. (I see 4x more iops on fio than on ceph, with same bandwith) for virtio-scsi num_queue = 8: ------------------------------- works with krbd and librbd -for random read : I jump from 7000 to 12000iops -for sequential, qemu aggreate reads in bigger ceph reads. (Same, I see 4xmore iops on fio than on ceph, with same bandwith). So it seem to be useful for some specific workloads. 2 ) Now, about cpu usage, it's seem than librbd use really more cpu than krbd, on host, librbd use 4x more cpu than krbd on qemu, librbd use 2x more cpu than krbd So, what could explain so much difference between both ? Regards, Alexandre fio iops seq read summary results ---------------------------------- qemu virtio iothread krbd vs qemu virtio iothread librbd : 27000 iops vs 15000 ipos qemu virtio krbd vs qemu virtio librbd : 19000 iops vs 15000 iops qemu virtio-scsi krbd vs qemu virtio librbd : 50000 iops vs 48000 iops host krbd vs host librbd : 36000 iops vs 25000 iops fio iops randread summary results ------------------------------ qemu virtio iothread krbd vs qemu virtio iothread librbd : 15000 iops vs 14000 iops qemu virtio krbd vs qemu virtio librbd : 14000 iops vs 15000 iops qemu virtio-scsi krbd vs qemu virtio librbd : 7500 iops vs 12000 iops host krbd vs host librbd : 38000 iops vs 25000 iops cpu usage ratio summary ------------------------ qemu virtio krbd vs qemu virtio librbd : 2x more cpu usage for librbd qemu virtio-scsi krbd vs qemu virtio-scsi librbd : 2x more cpu usage for librbd host krbd vs host librbd : 4x more cpu usage for fio-rbd RESULTS ------- host + fio - krbd ------------------- read ----- fio iops : 142.9MB/Ss , 36000 iops ceph : 134 MB/s rd, 34319 op/s fio : 70,4% kworker : 93,9% cpu = 164% cpu 100%cpu : 21000iops randread -------- fio: 151MB/S,38000 iops ceph :148 MB/s rd, 37932 op/s fio : 80%cpu kwoker : 110,3%cpu = 180% cpu 100%cpu : 21000 iops host + fio-rbdengin : --------------------- randread (cpu bound) -------------------- fio iops : 25000 ops ceph iops : 99636 kB/s rd, 24909 op/s fio : 460%cpu 100%cpu : 5415iops read (cpu bound) ----------------- fio iops : 25000 ops ceph ios : 94212 kB/s rd, 23553 op/s fio : 480%cpu 100%cpu : 5323iops qemu + krbd + virtio + iothread --------------------------------- read ---- fio :107MB/S : 27000 iops >>>SEEM THAT QEMU AGGREGATE READS ops ceph : 93942 kB/s rd, 12430 op/s kvm: 130% cpu - kworker : 41,2% = 171,2% cpu 100%cpu ratio : 7260iops randread -------- fio : 60MBS - 15000 iops ceph : 54400 kB/s rd, 13600 op/s kvm: 95,0% cpu - kworker : 42,1 % cpu = 137,1%cpu 100%cpu ratio : 9919 iops qemu + krbd + virtio ---------------------- read ----- fio : 70mbs/ , 19000iops ceph:75705 kB/s rd, 18926 op/s kvm : 164% cpu - kworker : 48,5% cpu = 212,5%cpu 100%cpu ratio : 8906 iops randread -------- fio : 54mbs/ , 14000iops ceph : 54800 kB/s rd, 13700 op/s kvm: 103% cpu - kworker 41,2% cpu = 144,2%cpu 100%cpu ratio : 9513 iops qemu + krbd + virtio-scsi (num_queue 8) -------------------------------------- read: ---- fio : 200MB / 50000 iops >>>SEEM THAT QEMU AGGREGATE READS ops ceph : 205 MB/s rd, 7648 op/s kvm: 145% kworker : 46,5% = 191,5%cpu 100%cpu ratio : 3993 iops randread: ---------- fio : 30MB/S / 7500 iops ceph : 29318 kB/s rd, 7329 op/s kvm : 150% kworker : 21,4% cpu = 171,4% cpu 100%cpu ratio : 4275 iops qemu + librbd + virtio + iothread ---------------------------------- read ---- fio : 60MBS/s , 15000iops ceph: 56199 kB/s rd, 14052 op/s kvm: 300% cpu 100%cpu : 4666iops randread --------- fio : 56MBS/s, 14000iops ceph : 55916 kB/s rd, 13979 op/s kvm: 300% cpu 100%cpu : 4659 iops qemu + librbd + virtio ------------------------- read ----- fio : 60MBS/s, 15000iops ceph : 63021 kB/s rd, 15755 op/s kvm: 300% cpu 100%cpu : 5233 iops randread -------- fio : 60MBS/s, 15000iops ceph : 55916 kB/s rd, 13979 op/s kvm : 300%cpu 100%cpu : 4659 iops qemu + librbd + virtio-scsi (num_queue 8) ---------------------------------------- read ---- fio : 256 MB/S , 48000iops >>>SEEM THAT QEMU AGGREGATE READS ops ceph : 244 MB/s rd, 12002 op/s kvm : 300% cpu 100%cpu : 4000 iops randread -------- fio: 12000iops ceph iops : 47511 kB/s rd, 11877 op/s kvm: 300% cpu 100%cpu: 3959 iops -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
