On Wed, Dec 05, 2012 at 01:32:15AM +0800, yi li wrote:
> My test:
>
> 1:guest os has 2 vcpu and has a virtio netcard, also host os just has
> 2 cpu cores, and do not do any smp_affinity
>
What about running the same test but on 32 host cpus. I expect result to
be much different.
> 2:using the jprobe to probe the function kvm_vcpu_kick on the host
> .......
> static int i=0;
> int jkvm_vcpu_kick(struct kvm_vcpu *vcpu)
> {
> int cpu = vcpu->cpu;
> printk("vcpu->cpu is %d\n",cpu);
> printk("the pcpu to handle the inject irq is %d\n", smp_processor_id());
> if(cpu != smp_processor_id())
> {
> i = i+ 1;
> }
> printk("the ipi sum is %d\n", i);
> /* Always end with a call to jprobe_return(). */
> jprobe_return();
> return 0;
> }
> ..........................
>
> run the command ping the guestos virtio netcard
>
> 3: test result,
> sometimes, it is very bad, you see blow, there has 14 ipi (64-50)
>
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.009937] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.009940] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.009942] the ipi sum is 50
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010008] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010011] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010013] the ipi sum is 50
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010051] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010052] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010055] the ipi sum is 50
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010093] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010094] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.010096] the ipi sum is 50
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.035829] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.035831] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.035833] the ipi sum is 50
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106226] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106229] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106232] the ipi sum is 51
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106309] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106311] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106313] the ipi sum is 51
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106359] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106360] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106361] the ipi sum is 51
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106385] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106386] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.106387] the ipi sum is 51
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129867] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129869] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129871] the ipi sum is 52
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129937] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129939] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129942] the ipi sum is 52
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129979] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129981] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.129983] the ipi sum is 52
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.130024] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.130025] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.130026] the ipi sum is 52
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.137952] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.137954] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.137956] the ipi sum is 53
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138022] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138024] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138026] the ipi sum is 53
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138064] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138066] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138068] the ipi sum is 53
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138106] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138107] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.138108] the ipi sum is 53
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145844] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145846] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145848] the ipi sum is 54
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145911] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145913] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145915] the ipi sum is 54
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145952] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145954] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145956] the ipi sum is 54
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145993] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.145995] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.146002] the ipi sum is 54
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187846] vcpu->cpu is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187848] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187850] the ipi sum is 54
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187898] vcpu->cpu is 0
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187900] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:52 liyi-ThinkPad-X200 kernel: [11470.187902] the ipi sum is 55
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.265951] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.265954] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.265956] the ipi sum is 56
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266037] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266039] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266041] the ipi sum is 56
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266082] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266083] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266084] the ipi sum is 56
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266109] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266110] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.266111] the ipi sum is 56
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274248] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274250] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274252] the ipi sum is 57
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274318] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274320] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274322] the ipi sum is 57
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274370] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274372] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274374] the ipi sum is 57
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274413] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274414] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.274415] the ipi sum is 57
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.281978] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.281981] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.281983] the ipi sum is 58
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282046] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282048] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282050] the ipi sum is 58
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282100] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282102] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282104] the ipi sum is 58
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282144] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282145] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.282146] the ipi sum is 58
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289860] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289862] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289864] the ipi sum is 59
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289929] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289931] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289933] the ipi sum is 59
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289972] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289974] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.289976] the ipi sum is 59
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.290018] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.290020] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.290021] the ipi sum is 59
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.305924] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.305926] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.305928] the ipi sum is 60
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.305997] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306000] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306002] the ipi sum is 60
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306040] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306042] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306044] the ipi sum is 60
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306082] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306083] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.306085] the ipi sum is 60
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.496784] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.496787] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.496790] the ipi sum is 61
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.498717] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.498719] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.498720] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499806] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499808] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499810] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499844] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499847] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.499849] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503811] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503814] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503816] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503848] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503850] the pcpu to
> handle the inject irq is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.503852] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.507808] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.507811] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.507813] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.511805] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.511807] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.511809] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.515807] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.515809] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.515811] the ipi sum is 62
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.596396] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.596399] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.596402] the ipi sum is 63
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687796] vcpu->cpu is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687799] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687801] the ipi sum is 63
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687862] vcpu->cpu is 1
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687864] the pcpu to
> handle the inject irq is 0
> Dec 5 00:04:53 liyi-ThinkPad-X200 kernel: [11470.687866] the ipi sum is 64
>
> 4: the above result just happen once, when i reboot the host, the IPI
> is not increase occured by kvm_vcpu_kick, i will check it more detail.
>
> YiLi
> Thanks
>
>
> 2012/12/4 Gleb Natapov <gleb@xxxxxxxxxx>:
> > On Tue, Dec 04, 2012 at 11:08:41AM +0800, yi li wrote:
> >> The cpu inject the interrupt to vcpu which vcpu->cpu is the same as it.
> >> And it maybe avoid a IPI between the cpu core.
> >>
> > How often this happens in your testing? Regardless we cannot disable
> > preemption while iterating over all vcpus. Low latency people will kill
> > us.
> >
> >> Signed-off-by: Yi Li <yilikernel@xxxxxxxxx>
> >>
> >> --- linux/virt/kvm/irq_comm.c 2012-12-04 10:14:57.711024619 +0800
> >> +++ linux/virt/kvm/irq_comm.c 2012-12-04 11:01:27.728859597 +0800
> >> @@ -64,9 +64,10 @@ inline static bool kvm_is_dm_lowest_prio
> >> int kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src,
> >> struct kvm_lapic_irq *irq)
> >> {
> >> - int i, r = -1;
> >> + int i, cpu, r = -1;
> >> struct kvm_vcpu *vcpu, *lowest = NULL;
> >>
> >> + cpu = get_cpu();
> >> if (irq->dest_mode == 0 && irq->dest_id == 0xff &&
> >> kvm_is_dm_lowest_prio(irq)) {
> >> printk(KERN_INFO "kvm: apic: phys broadcast and lowest prio\n");
> >> @@ -89,13 +90,17 @@ int kvm_irq_delivery_to_apic(struct kvm
> >> r = 0;
> >> r += kvm_apic_set_irq(vcpu, irq);
> >> } else if (kvm_lapic_enabled(vcpu)) {
> >> - if (!lowest)
> >> + if(vcpu->cpu == cpu) {
> >> + lowest = vcpu;
> >> + break;
> >> + }
> >> + else if (!lowest)
> >> lowest = vcpu;
> >> else if (kvm_apic_compare_prio(vcpu, lowest) < 0)
> >> lowest = vcpu;
> >> }
> >> }
> >> -
> >> + put_cpu();
> >> if (lowest)
> >> r = kvm_apic_set_irq(lowest, irq);
> >>
> >>
> >> YiLi
> >> Thanks
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> > Gleb.
--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
