Re: [performance] rbd kernel module versus qemu librbd

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi, 

New information, i think the poor performance is due to too many threads of qume-system-x86 process.

For normal case, it just use about 200 threads.

For abnormal case, it will use about 400 threads or 700 threads, and the performance is:

200 threads > 400 threads > 700 threads

Now, i guess it performance down is due to the competition between the threads. As you could see, i paste the
perf record before. 

The problem is really stuck us.....

So, anyone know why the threads number of qume-system-x86 increasing?
And any way we could control it.

Thanks!


hzwulibin@xxxxxxxxx
 
From: hzwulibin@xxxxxxxxx
Date: 2015-10-23 13:15
To: Alexandre DERUMIER
CC: ceph-users
Subject: Re: Re: [ceph-users] [performance] rbd kernel module versus qemu librbd
Yeah, you are right.  Test the rbd volume form host is fine.

Now, at least we could affirm ti's the qemu or kvm problem, not ceph.


hzwulibin@xxxxxxxxx
 
Date: 2015-10-23 12:51
Subject: Re: [performance] rbd kernel module versus qemu librbd
>>Anyway, i could try to collect somthing, maybe there are some clues.
 
And you don't have problem to read/write to this rbd from host with fio-rbd ?
(try a read full the rbd volume for example)
 
----- Mail original -----
De: hzwulibin@xxxxxxxxx
À: "aderumier" <aderumier@xxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Envoyé: Vendredi 23 Octobre 2015 06:42:41
Objet: Re: Re: [performance] rbd kernel module versus qemu librbd
 
Oh, no, from the phenomenon. IO in VM is wait for the host to completion.
The CPU wait in VM is very high.
Anyway, i could try to collect somthing, maybe there are some clues.
 
 
hzwulibin@xxxxxxxxx
 
 
 
From: Alexandre DERUMIER
Date: 2015-10-23 12:39
To: hzwulibin
CC: ceph-users
Subject: Re: [performance] rbd kernel module versus qemu librbd
Do you have tried to use perf inside the faulty guest too ?
----- Mail original -----
De: hzwulibin@xxxxxxxxx
À: "aderumier" <aderumier@xxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Envoyé: Vendredi 23 Octobre 2015 06:15:07
Objet: Re: Re: [ceph-users] [performance] rbd kernel module versus qemu librbd
btw, we use perf to track the process qemu-system-x86(15801), there is an abnormal function:
Samples: 1M of event 'cycles', Event count (approx.): 1057109744252
- 75.23% qemu-system-x86 [kernel.kallsyms] [k] do_raw_spin_lock
- do_raw_spin_lock
+ 54.44% 0x7fc79fc769d9 + 45.31% 0x7fc79fc769ab
So, maybe it's the kvm problem?
hzwulibin@xxxxxxxxx
From: hzwulibin@xxxxxxxxx
Date: 2015-10-23 11:54
To: Alexandre DERUMIER
CC: ceph-users
Subject: Re: Re: [performance] rbd kernel module versus qemu librbd
Hi, list
We still stuck on this problem, when this problem comes, the CPU usage of qemu-system-x86 if very high(1420):
15801 libvirt- 20 0 33.7g 1.4g 11m R 1420 0.6 1322:26 qemu-system-x86
quem-system-x86 process 15801 is responsible for the VM.
Anyone has ever run into this problem also.
hzwulibin@xxxxxxxxx
BQ_BEGIN
From: hzwulibin@xxxxxxxxx
Date: 2015-10-22 10:15
To: Alexandre DERUMIER
CC: ceph-users
Subject: Re: Re: [ceph-users] [performance] rbd kernel module versus qemu librbd
Hi,
Sure, all those could help, but not so much -:)
Now, we find it's the VM problem. CPU on the host is very high.
We create a new VM could solve this problem, but don't know why until now.
Here is the detail version info:
Compiled against library: libvirt 1.2.9
Using library: libvirt 1.2.9
Using API: QEMU 1.2.9
Running hypervisor: QEMU 2.1.2
Are there any already know bugs about those version?
Thanks!
hzwulibin@xxxxxxxxx
BQ_BEGIN
From: Alexandre DERUMIER
Date: 2015-10-21 18:38
To: hzwulibin
CC: ceph-users
Subject: Re: [performance] rbd kernel module versus qemu librbd
here a libvirt sample to enable iothreads:
<domain>
<iothreads>2</iothreads>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' iothread='1'/>
<source file='/var/lib/libvirt/images/iothrtest1.img'/>
<target dev='vda' bus='virtio'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' iothread='2'/>
<source file='/var/lib/libvirt/images/iothrtest2.img'/>
<target dev='vdb' bus='virtio'/>
</disk>
</domain>
With this, you can scale with multiple disks. (but it should help a little bit with 1 disk too)
----- Mail original -----
De: hzwulibin@xxxxxxxxx
À: "aderumier" <aderumier@xxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Envoyé: Mercredi 21 Octobre 2015 10:31:56
Objet: Re: [performance] rbd kernel module versus qemu librbd
Hi,
let me post the version and configuration here first.
host os: debian 7.8 kernel: 3.10.45
guest os: debian 7.8 kernel: 3.2.0-4
qemu version:
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2.1~bpo70+1 all PXE boot firmware - ROM images for qemu
ii qemu-kvm 1:2.1+dfsg-12~bpo70+1 amd64 QEMU Full virtualization on x86 hardware
ii qemu-system-common 1:2.1+dfsg-12~bpo70+1 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 1:2.1+dfsg-12~bpo70+1 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.1+dfsg-12~bpo70+1 amd64 QEMU utilities
vm config:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='cinder'>
<secret type='ceph' uuid='****'/>
</auth>
<source protocol='rbd' name='*****'>
<host name='***' port='6789'/>
<host name='***' port='6789'/>
<host name='***' port='6789'/>
</source>
<target dev='vdf' bus='virtio'/>
<serial>*******</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
</disk>
Thanks!
hzwulibin@xxxxxxxxx
From: Alexandre DERUMIER
Date: 2015-10-21 14:01
To: hzwulibin
CC: ceph-users
Subject: Re: [performance] rbd kernel module versus qemu librbd
Damn, that's a huge difference.
What is your host os, guest os , qemu version and vm config ?
As an extra boost, you could enable iothread on virtio disk.
(It's available on libvirt but not on openstack yet).
If it's a test server, maybe could you test it with proxmox 4.0 hypervisor
https://www.proxmox.com
I have made a lot of patch inside it to optimize rbd (qemu+jemalloc, iothreads,...)
----- Mail original -----
De: hzwulibin@xxxxxxxxx
À: "aderumier" <aderumier@xxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxx>
Envoyé: Mercredi 21 Octobre 2015 06:11:20
Objet: Re: Re: [performance] rbd kernel module versus qemu librbd
Hi,
Thanks for you reply.
I do more test here and things change more strange, now i only could get about 4k iops in VM:
1. use fio with ioengine rbd to test the volume on the real machine
[global]
ioengine=rbd
clientname=admin
pool=vol_ssd
rbdname=volume-4f4f9789-4215-4384-8e65-127a2e61a47f
rw=randwrite
bs=4k
group_reporting=1
[rbd_iodepth32]
iodepth=32
[rbd_iodepth1]
iodepth=32
[rbd_iodepth28]
iodepth=32
[rbd_iodepth8]
iodepth=32
could achive about 18k iops.
2. test the same volume in VM, achive about 4.3k iops
[global]
rw=randwrite
bs=4k
ioengine=libaio
#ioengine=sync
iodepth=128
direct=1
group_reporting=1
thread=1
filename=/dev/vdb
[task1]
iodepth=32
[task2]
iodepth=32
[task3]
iodepth=32
[task4]
iodepth=32
Using cep osd perf to check the osd latency, all less than 1 ms.
Using iostat to check the osd %util, about 10 in case 2 test.
Using dstat to check VM status:
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 4 51 43 0 0| 0 17M| 997B 3733B| 0 0 |3476 6997
2 5 51 43 0 0| 0 18M| 714B 4335B| 0 0 |3439 6915
2 5 50 43 0 0| 0 17M| 594B 3150B| 0 0 |3294 6617
1 3 52 44 0 0| 0 18M| 648B 3726B| 0 0 |3447 6991
1 5 51 43 0 0| 0 18M| 582B 3208B| 0 0 |3467 7061
Finally, using iptraf to check the package size in the VM, almost packages's
size are around 1 to 70 and 71 to 140 bytes. That's different from real machine.
But maybe iptraf on the VM can't prove anything, i check the real machine which the VM located on.
It seems no abnormal.
BTW, my VM is located on the ceph storage node.
Anyone can give me more sugestions?
Thanks!
hzwulibin@xxxxxxxxx
From: Alexandre DERUMIER
Date: 2015-10-20 19:36
To: hzwulibin
CC: ceph-users
Subject: Re: [performance] rbd kernel module versus qemu librbd
Hi,
I'm able to reach around same performance with qemu-librbd vs qemu-krbd,
when I compile qemu with jemalloc
(http://git.qemu.org/?p=qemu.git;a=commit;h=7b01cb974f1093885c40bf4d0d3e78e27e531363)
on my test, librbd with jemalloc still use 2x more cpu than krbd,
so cpu could be bottleneck too.
with fasts cpu (3.1ghz), I'm able to reach around 70k iops 4k with rbd volume, both with krbd or librbd
----- Mail original -----
De: hzwulibin@xxxxxxxxx
À: "ceph-users" <ceph-users@xxxxxxxx>
Envoyé: Mardi 20 Octobre 2015 10:22:33
Objet: [performance] rbd kernel module versus qemu librbd
Hi,
I have a question about the IOPS performance for real machine and virtual machine.
Here is my test situation:
1. ssd pool (9 OSD servers with 2 osds on each server, 10Gb networks for public & cluster networks)
2. volume1: use rbd create a 100G volume from the ssd pool and map to the real machine
3. volume2: use cinder create a 100G volume form the ssd pool and atach to a guest host
4. disable rbd cache
5. fio test on the two volues:
[global]
rw=randwrite
bs=4k
ioengine=libaio
iodepth=64
direct=1
size=64g
runtime=300s
group_reporting=1
thread=1
volume1 got about 24k IOPS and volume got about 14k IOPS.
We could see performance of volume2 is not good compare to volume1, so is it normal behabior of guest host?
If not, what maybe the problem?
Thanks!
hzwulibin@xxxxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
BQ_END
 
BQ_END
 
 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux