>Hi, indeed qemu use a single thread per queue. >I think it's not a problem with common storage (nfs,scsi,..) because they use less cpu ressource than librbd, so you can reach >easily > 100000iops with 1thread. I think it's a common problem shared by all storage backend(nfs,scsi), but Ceph take longer time to sent out an IO(30us), while NFS and scsi is really simple(no crush, no stripping, etc) that may only take 3us to send out an IO, so the upper bound is 10X than ceph. >Now, the good new, virtio-scsi support multiqueue , with num_queues option. >I have done bench with num_queues, and now I can finally reach 50000iops with a single qemu disk-> rbd volume :) >If you use libvirt, here the config: >http://www.redhat.com/archives/libvir-list/2013-April/msg00021.html Thanks, this is really very helpful for us, we will try it out, thank you -----Original Message----- From: Alexandre DERUMIER [mailto:aderumier@xxxxxxxxx] Sent: Thursday, October 16, 2014 2:25 PM To: Chen, Xiaoxi Cc: ceph-devel@xxxxxxxxxxxxxxx; Mark Nelson Subject: Re: 10/14/2014 Weekly Ceph Performance Meeting >>We also find this before, seems it's because QEMU use single thread for IO, I tried to enable the debug log of in librbd, find that the threaded is always the same. Assuming the backend is power enough, so how >>many IOs can be sent out by qemu == how many IOPS can we get. >> >>The upper bound may varies depends on how fast QEMU can send out a request, I remember we have tried different CPU model and get best with an very old NHM CPU(with high frequency to 2.93Ghz, take ~0.03 to send >>out a request and we get 22K read IOPS bound), better than new SNB CPU. Frequency do play important part since it's single thread. Hi, indeed qemu use a single thread per queue. I think it's not a problem with common storage (nfs,scsi,..) because they use less cpu ressource than librbd, so you can reach easily > 100000iops with 1thread. Now, the good new, virtio-scsi support multiqueue , with num_queues option. I have done bench with num_queues, and now I can finally reach 50000iops with a single qemu disk-> rbd volume :) If you use libvirt, here the config: http://www.redhat.com/archives/libvir-list/2013-April/msg00021.html Now, about fio-rbd cpu usage, something is really strange, fio (rbd engine) on qemu host : 40000iops - 8 cores 100% fio (aio) inside qemu with virtio-scsi + num_queues : 50000 iops - around 4 cores 100% So, maybe something is bad in fio-rbd implementation ? I need to bench again to be sure about this, I'll send results in some days. ----- Mail original ----- De: "Xiaoxi Chen" <xiaoxi.chen@xxxxxxxxx> À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>, "Mark Nelson" <mark.nelson@xxxxxxxxxxx> Cc: ceph-devel@xxxxxxxxxxxxxxx Envoyé: Jeudi 16 Octobre 2014 05:27:13 Objet: RE: 10/14/2014 Weekly Ceph Performance Meeting We also find this before, seems it's because QEMU use single thread for IO, I tried to enable the debug log of in librbd, find that the threaded is always the same. Assuming the backend is power enough, so how many IOs can be sent out by qemu == how many IOPS can we get. The upper bound may varies depends on how fast QEMU can send out a request, I remember we have tried different CPU model and get best with an very old NHM CPU(with high frequency to 2.93Ghz, take ~0.03 to send out a request and we get 22K read IOPS bound), better than new SNB CPU. Frequency do play important part since it's single thread. -----Original Message----- From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Alexandre DERUMIER Sent: Wednesday, October 15, 2014 11:55 PM To: Mark Nelson Cc: ceph-devel@xxxxxxxxxxxxxxx Subject: Re: 10/14/2014 Weekly Ceph Performance Meeting > 2) in qemu, it's impossible to reach more than around 7000iops with 1 disk. (maybe is it also related to cpu or threads number). > I have also try with the new qemu iothread/dataplane feature, but it doesn't help. >>If its 1 volume, does adding another volume on the same VM help? >>>As far I remember, yes . I'll test again to confirm. I have done test, It's scaling with multiple virtio disk on multiple rbd volume. (Not sure, but maybe it's related to iodepth bug showed in this meeting in intel slides ?) ----- Mail original ----- De: "Alexandre DERUMIER" <aderumier@xxxxxxxxx> À: "Mark Nelson" <mark.nelson@xxxxxxxxxxx> Cc: ceph-devel@xxxxxxxxxxxxxxx Envoyé: Mercredi 15 Octobre 2014 16:42:04 Objet: Re: 10/14/2014 Weekly Ceph Performance Meeting >>Sure! Please feel free to add this or other topics that are >>useful/interesting to the etherpad. Please include your name though so >>we know who's brought it up. Even if we don't get to everything it >>will provide useful topics for the subsequent weeks. Ok,great,I'll do it. > > Currently I see 2 performance problems with librbd: > > 1) The cpu usage is quite huge. (I'm cpu bound with 8cores CPU E5-2603 > v2 @ 1.80GHz, with 40000iops 4k read using fio-rbd) >>Interesting. Have you taken a look with perf or other tools to see >>where time is being spent? Not yet, but I can try to do it, I'll have time next week. > > 2) in qemu, it's impossible to reach more than around 7000iops with 1 disk. (maybe is it also related to cpu or threads number). > I have also try with the new qemu iothread/dataplane feature, but it doesn't help. >>1 disk meaning 1 OSD, or 1 disk meaning 1 volume on a VM? yes, 1 disk = 1 volume on VM >>If its 1 volume, does adding another volume on the same VM help? As far I remember, yes . I'll test again to confirm. Note that when benching with fio-rbd, I need to increase the client number too. (1client - queue depth 32: +- 8000iops 2clients - queue depth 32: +- 16000iops .... ) So maybe it's related >>I'm not familiar with the new qemu options, so that would be good to >>discuss at the meeting too! The dataplane/iothread feature allow virtio disk to reach around 1.000.000 iops vs 100.000 iops without dataplane http://www.linux-kvm.org/wiki/images/1/17/Kvm-forum-2013-Effective-multithreading-in-QEMU.pdf Syntax to enable it: qemu -object iothread,id=iothread0 -device virtio-blk-pci,iothread=iothread0,.... Regards, Alexandre ----- Mail original ----- De: "Mark Nelson" <mark.nelson@xxxxxxxxxxx> À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>, "Mark Nelson" <mark.nelson@xxxxxxxxxxx> Cc: ceph-devel@xxxxxxxxxxxxxxx Envoyé: Mercredi 15 Octobre 2014 15:26:20 Objet: Re: 10/14/2014 Weekly Ceph Performance Meeting On 10/15/2014 01:22 AM, Alexandre DERUMIER wrote: > Hi, > > about performance, maybe could it be great to also include client side performance ? Sure! Please feel free to add this or other topics that are useful/interesting to the etherpad. Please include your name though so we know who's brought it up. Even if we don't get to everything it will provide useful topics for the subsequent weeks. > > Currently I see 2 performance problems with librbd: > > 1) The cpu usage is quite huge. (I'm cpu bound with 8cores CPU E5-2603 v2 @ 1.80GHz, with 40000iops 4k read using fio-rbd) Interesting. Have you taken a look with perf or other tools to see where time is being spent? > > 2) in qemu, it's impossible to reach more than around 7000iops with 1 disk. (maybe is it also related to cpu or threads number). > I have also try with the new qemu iothread/dataplane feature, but it doesn't help. 1 disk meaning 1 OSD, or 1 disk meaning 1 volume on a VM? If its 1 volume, does adding another volume on the same VM help? I'm not familiar with the new qemu options, so that would be good to discuss at the meeting too! > > > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
