Hi , yes librbd can use a lot of cpu, mainly if you want to do a lot of iops. I have benched it vs krbd driver, and even with tcmalloc/jemalloc fio librbd use twice more cpu than krbd. as qemu use only 1 thread by default for all disks, this is a bottleneck. using multiple iothreads (1by disk) in qemu, will increase speed. >From my bench, I can't get more than 45K iops (4k randread) with 1 qemu thread. with cephx, debug disabled. rbd protocol is more complex than a simple protocol like nfs. The crush algorithm is running on client side for example. ----- Mail original ----- De: "Nikola Ciprich" <nikola.ciprich@xxxxxxxxxxx> À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> Envoyé: Mardi 16 Juin 2015 10:45:12 Objet: qemu (or librbd in general) - very high load on client side Hello dear ceph developers and users, I've spent some time tuning and measuring our ceph cluster performance, and noticed quite strange thing.. I've been using fio (using both rbd engine on hosts and direct block (aio) engine inside qemu-kvm guests (qemu connected to ceph storage using rbd)) and I noticed client part always generates huge amount of CPU load and therefore CLIENT seems to be the bottleneck. For example, when I measure direct SSD performance on one of ceph OSDs, I'm getting 100k IOPS (which is OK, according to SSD specs) using fio, but when I measure performance of ceph SSD pool volume, it's much worse. I'd understand, if the bottlenect would be ceph-osd processes (or some other ceph component), but it seems to me fio using rbd engine is the problem here (it's able to eat 6 CPU cores itself). Seems to be very similar when using qemu to access the ceph storage - it shows very high cpu utilisation (i'm using virtio-scsi for guest disk emulation). This behaviour is for both random and sequential IO. preloading libtcmalloc helps fio (and I also tried compiling qemu with libtcmallc, it also helps), but still it seems to me that there could be something wrong in librbd.. Has anyone else noticed this behaviour? I noticed on some mail threads, that disabling cephx authentication can help a lot, but I don't really like this idea and haven't tried it yet.. with best regards nik -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@xxxxxxxxxxx ------------------------------------- _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |