On Tue, Nov 23, 2010 at 10:53:10AM +0800, lidong chen wrote:
> Test method:
> Send the same traffic load between virtio_pci MSI-X disable and
> enable,and compare the cpu rate of host os.
> I used the same version of virtio driver, only modify the msi-x option.
> the host os version is 2.6.32.
> the virtio dirver is from rhel6.
> the guest version os is 2.6.16.
>
> Test result:
> with msi-x disable, the cpu rate of host os is 110%.
> with msi-x enable, the cpu rate of host os is 140%.
>
> the /proc/interrupt with msi-x disable is below:
> CPU0 CPU1
> 0: 12326706 0 IO-APIC-edge timer
> 1: 8 0 IO-APIC-edge i8042
> 8: 0 0 IO-APIC-edge rtc
> 9: 0 0 IO-APIC-level acpi
> 10: 4783008 0 IO-APIC-level virtio2, virtio3
> 11: 5363828 0 IO-APIC-level virtio1, virtio4, virtio5
> 12: 104 0 IO-APIC-edge i8042
> NMI: 2857871 2650796
> LOC: 12324952 12325609
> ERR: 0
> MIS: 0
>
> the /proc/interrupt with msi-x enable is below:
> CPU0 CPU1
> 0: 1896802 0 IO-APIC-edge timer
> 1: 8 0 IO-APIC-edge i8042
> 4: 14 0 IO-APIC-edge serial
> 8: 0 0 IO-APIC-edge rtc
> 9: 0 0 IO-APIC-level acpi
> 10: 0 0 IO-APIC-level virtio1, virtio2, virtio5
> 11: 1 0 IO-APIC-level virtio0, virtio3, virtio4
This one probably means there's a bug: when msix
is enabled there should not be any level interrupts.
> 12: 104 0 IO-APIC-edge i8042
> 50: 1 0 PCI-MSI-X virtio2-output
> 58: 0 0 PCI-MSI-X virtio3-config
> 66: 2046985 0 PCI-MSI-X virtio3-input
> 74: 2 0 PCI-MSI-X virtio3-output
> 82: 0 0 PCI-MSI-X virtio4-config
> 90: 217 0 PCI-MSI-X virtio4-input
> 98: 0 0 PCI-MSI-X virtio4-output
> 177: 0 0 PCI-MSI-X virtio0-config
> 185: 341831 0 PCI-MSI-X virtio0-input
> 193: 1 0 PCI-MSI-X virtio0-output
> 201: 0 0 PCI-MSI-X virtio1-config
> 209: 188747 0 PCI-MSI-X virtio1-input
> 217: 1 0 PCI-MSI-X virtio1-output
> 225: 0 0 PCI-MSI-X virtio2-config
> 233: 2204149 0 PCI-MSI-X virtio2-input
> NMI: 1455767 1426226
> LOC: 1896099 1896637
> ERR: 0
> MIS: 0
I just noticed that above msi-x shows 4M interrupts
and 1.5M NMI but non-MSI shows 10M and 3M.
> Code difference:
> I disalbe msi-x by modify the function vp_find_vqs like this:
You can simply supply nvectors=0 in qemu.
> static int vp_find_vqs(struct virtio_device *vdev, unsigned nvqs,
> struct virtqueue *vqs[],
> vq_callback_t *callbacks[],
> const char *names[])
> {
>
> #if 0
> int err;
>
> /* Try MSI-X with one vector per queue. */
> err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names, true, true);
> if (!err)
> return 0;
> /* Fallback: MSI-X with one vector for config, one shared for queues. */
> err = vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
> true, false);
> if (!err)
> return 0;
> /* Finally fall back to regular interrupts. */
> #endif
>
> return vp_try_to_find_vqs(vdev, nvqs, vqs, callbacks, names,
> false, false);
> }
>
> Conclusion:
> msi-x enable waste more cpu resource is caused by MSIX mask bit. In
> older kernels program this bit twice
> on every interrupt. and caused ept violation.
Wait a second, older kernels don't have msix support in virtio,
do they?
> So I think we should add a param to control this.with older kernels,
> we should disable MSIX.
> And I think this should deal by qemu.
I would like to see a comparison of msix enabled and disabled
with a guest that supports msix natively.
--
MST
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
