On 11/12/2012 07:33 AM, Stefan Priebe - Profihost AG wrote:
Adding this to ceph.conf on kvm host adds another 2000 iops (20.000
iop/s with one VM). I'm sure most of them are useless on a client kvm /
rbd host but i don't know which one makes sense ;-)
[global]
debug ms = 0/0
debug rbd = 0/0
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
[client]
For the client side you'd these settings to disable all debug logging:
[client]
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug filer = 0/0
debug objecter = 0/0
debug rados = 0/0
debug rbd = 0/0
debug objectcacher = 0/0
debug client = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
Josh
debug ms = 0/0
debug rbd = 0/0
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
Stefan
Am 12.11.2012 15:35, schrieb Alexandre DERUMIER:
Another idea,
do you have tried to put
debug lockdep = 0/0
debug context = 0/0
debug crush = 0/0
debug buffer = 0/0
debug timer = 0/0
debug journaler = 0/0
debug osd = 0/0
debug optracker = 0/0
debug objclass = 0/0
debug filestore = 0/0
debug journal = 0/0
debug ms = 0/0
debug monc = 0/0
debug tp = 0/0
debug auth = 0/0
debug finisher = 0/0
debug heartbeatmap = 0/0
debug perfcounter = 0/0
debug asok = 0/0
debug throttle = 0/0
in a ceph.conf on your kvm host ?
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>
À: "Stefan Priebe - Profihost AG" <s.priebe@xxxxxxxxxxxx>
Cc: pve-devel@xxxxxxxxxxxxxxx
Envoyé: Lundi 12 Novembre 2012 15:26:36
Objet: Re: [pve-devel] less cores more iops / speed
Maybe some tracing on kvm process could give us clues to find where
the cpu is used ?
Also another idea, can you try with "auth supported=none" ? maybe they
are some overhead with ceph authenfication ?
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>
À: "Stefan Priebe - Profihost AG" <s.priebe@xxxxxxxxxxxx>
Cc: pve-devel@xxxxxxxxxxxxxxx
Envoyé: Lundi 12 Novembre 2012 15:20:07
Objet: Re: [pve-devel] less cores more iops / speed
Ok thanks.
Seem to use a lot of cpu vs nfs,iscsi ...
I hope that ceph dev will work on this soon !
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.priebe@xxxxxxxxxxxx>
À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>
Cc: "eric" <eric@xxxxxxxxxxx>, pve-devel@xxxxxxxxxxxxxxx
Envoyé: Lundi 12 Novembre 2012 15:05:08
Objet: Re: [pve-devel] less cores more iops / speed
Am 12.11.2012 13:49, schrieb Alexandre DERUMIER:
One VM on one Host: 18.000 IOP/s
Two VM on one Host: 2x11.000 IOP/s
Three VM on one Host: 3x7.000 IOP/s
And host cpu is 100% ?
No. For three VMs yes. For one and two no. I think librbd / rbd
implementation in kvm is the bottleneck here.
Stefan
----- Mail original -----
De: "Stefan Priebe - Profihost AG" <s.priebe@xxxxxxxxxxxx>
À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>
Cc: "eric" <eric@xxxxxxxxxxx>, pve-devel@xxxxxxxxxxxxxxx
Envoyé: Lundi 12 Novembre 2012 12:58:35
Objet: Re: [pve-devel] less cores more iops / speed
Am 12.11.2012 08:51, schrieb Alexandre DERUMIER:
Right now RBD in KVM is limited by CPU speed.
Good to known, so it's seem lack of threading, or maybe somes locks.
(so faster cpu give more iops).
If you lauch parallel fio on same host on different guest, do you
get more total iops ? (for me it's scale)
One VM on one Host: 18.000 IOP/s
Two VM on one Host: 2x11.000 IOP/s
Three VM on one Host: 3x7.000 IOP/s
if you launch 2 parallel fio, on same guest (on differents disk), do
you get more iops ? (for me, it doesn't scale, so raid0 in guest
doesn't help).
No it doesn't scale.
Stefan
----- Mail original -----
De: "Stefan Priebe" <s.priebe@xxxxxxxxxxxx>
À: "Alexandre DERUMIER" <aderumier@xxxxxxxxx>
Cc: "eric" <eric@xxxxxxxxxxx>, pve-devel@xxxxxxxxxxxxxxx
Envoyé: Dimanche 11 Novembre 2012 13:07:36
Objet: Re: [pve-devel] less cores more iops / speed
Am 11.11.2012 12:12, schrieb Alexandre DERUMIER:
If I remember good, stefan can achieve 100.000 iops with iscsi with
same kvm host.
Correct but this was always with scsi-generic and I/O multipathing on
host. rbd does not support scsi-generic ;-(
I have checked ceph mailing, stefan seem to have resolved his
problem with dual core with bios update !
Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on
Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz
Single Xeon vs. 2.5Ghz Dual Xeon.
Right now RBD in KVM is limited by CPU speed.
Greets,
Stefan
_______________________________________________
pve-devel mailing list
pve-devel@xxxxxxxxxxxxxxx
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@xxxxxxxxxxxxxxx
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
