i apply patch correctly.
the addr is not in mmio range because kvm_io_bus_write test the addr
for each device.
/* kvm_io_bus_write - called under kvm->slots_lock */
int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
int len, const void *val)
{
int i;
struct kvm_io_bus *bus = rcu_dereference(kvm->buses[bus_idx]);
for (i = 0; i < bus->dev_count; i++)
if (!kvm_iodevice_write(bus->devs[i], addr, len, val))
return 0;
return -EOPNOTSUPP;
}
the test result(cpu rate) before apply patch:
hostos guestos
CPU0 15.85 58.05
CPU1 2.97 70.56
CPU2 3.25 69.00
CPU3 3.31 68.59
CPU4 5.11 47.46
CPU5 4.70 29.28
CPU6 6.00 5.96
CPU7 4.75 2.58
the test result(cpu rate) after apply patch:
hostos guestos
CPU0 11.89 60.92
CPU1 2.18 68.07
CPU2 2.61 66.18
CPU3 2.44 66.46
CPU4 2.67 44.98
CPU5 2.31 26.81
CPU6 1.93 5.65
CPU7 1.79 2.14
the total cpu rate reduce from 397% to 369%.
i pin the vcpu below, and only vcpu0 handle msix interrupt.
virsh vcpupin brd2vm1sriov 0 0
virsh vcpupin brd2vm1sriov 1 1
virsh vcpupin brd2vm1sriov 2 2
virsh vcpupin brd2vm1sriov 3 3
virsh vcpupin brd2vm1sriov 4 4
virsh vcpupin brd2vm1sriov 5 5
virsh vcpupin brd2vm1sriov 6 6
virsh vcpupin brd2vm1sriov 7 7
I will test virtio_net with msix enable later, and compare to it with
msix disable.
to see which is better?
2010/12/2 Michael S. Tsirkin <mst@xxxxxxxxxx>:
> On Thu, Dec 02, 2010 at 07:52:00PM +0800, Sheng Yang wrote:
>> On Thu, Dec 2, 2010 at 5:49 PM, Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
>> > On Thu, Dec 02, 2010 at 09:13:28AM +0800, Yang, Sheng wrote:
>> >> On Wednesday 01 December 2010 22:03:58 Michael S. Tsirkin wrote:
>> >> > On Wed, Dec 01, 2010 at 04:41:38PM +0800, lidong chen wrote:
>> >> > > I used sr-iov, give each vm 2 vf.
>> >> > > after apply the patch, and i found performence is the same.
>> >> > >
>> >> > > the reason is in function msix_mmio_write, mostly addr is not in mmio
>> >> > > range.
>> >> > >
>> >> > > static int msix_mmio_write(struct kvm_io_device *this, gpa_t addr, int
>> >> > > len,
>> >> > >
>> >> > > á á á á á á á á á á áconst void *val)
>> >> > >
>> >> > > {
>> >> > >
>> >> > > á struct kvm_assigned_dev_kernel *adev =
>> >> > >
>> >> > > á á á á á á á á á container_of(this, struct kvm_assigned_dev_kernel,
>> >> > >
>> >> > > á á á á á á á á á á á á á á á ámsix_mmio_dev);
>> >> > >
>> >> > > á int idx, r = 0;
>> >> > > á unsigned long new_val = *(unsigned long *)val;
>> >> > >
>> >> > > á mutex_lock(&adev->kvm->lock);
>> >> > > á if (!msix_mmio_in_range(adev, addr, len)) {
>> >> > >
>> >> > > á á á á á // return here.
>> >> > >
>> >> > > á á á á á á á á ár = -EOPNOTSUPP;
>> >> > >
>> >> > > á á á á á goto out;
>> >> > >
>> >> > > á }
>> >> > >
>> >> > > i printk the value:
>> >> > > addr á á á á á á start á á á á á end á á á á á len
>> >> > > F004C00C á F0044000 áF0044030 á á 4
>> >> > >
>> >> > > 00:06.0 Ethernet controller: Intel Corporation Unknown device 10ed (rev
>> >> > > 01)
>> >> > >
>> >> > > á Subsystem: Intel Corporation Unknown device 000c
>> >> > > á Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> >> > >
>> >> > > Stepping- SERR- FastB2B-
>> >> > >
>> >> > > á Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> >> > >
>> >> > > <TAbort- <MAbort- >SERR- <PERR-
>> >> > >
>> >> > > á Latency: 0
>> >> > > á Region 0: Memory at f0040000 (32-bit, non-prefetchable) [size=16K]
>> >> > > á Region 3: Memory at f0044000 (32-bit, non-prefetchable) [size=16K]
>> >> > > á Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
>> >> > >
>> >> > > á á á á á Vector table: BAR=3 offset=00000000
>> >> > > á á á á á PBA: BAR=3 offset=00002000
>> >> > >
>> >> > > 00:07.0 Ethernet controller: Intel Corporation Unknown device 10ed (rev
>> >> > > 01)
>> >> > >
>> >> > > á Subsystem: Intel Corporation Unknown device 000c
>> >> > > á Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> >> > >
>> >> > > Stepping- SERR- FastB2B-
>> >> > >
>> >> > > á Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
>> >> > >
>> >> > > <TAbort- <MAbort- >SERR- <PERR-
>> >> > >
>> >> > > á Latency: 0
>> >> > > á Region 0: Memory at f0048000 (32-bit, non-prefetchable) [size=16K]
>> >> > > á Region 3: Memory at f004c000 (32-bit, non-prefetchable) [size=16K]
>> >> > > á Capabilities: [40] MSI-X: Enable+ Mask- TabSize=3
>> >> > >
>> >> > > á á á á á Vector table: BAR=3 offset=00000000
>> >> > > á á á á á PBA: BAR=3 offset=00002000
>> >> > >
>> >> > > +static bool msix_mmio_in_range(struct kvm_assigned_dev_kernel *adev,
>> >> > > + á á á á á á á á á á á gpa_t addr, int len)
>> >> > > +{
>> >> > > + gpa_t start, end;
>> >> > > +
>> >> > > + BUG_ON(adev->msix_mmio_base == 0);
>> >> > > + start = adev->msix_mmio_base;
>> >> > > + end = adev->msix_mmio_base + PCI_MSIX_ENTRY_SIZE *
>> >> > > + á á á á adev->msix_max_entries_nr;
>> >> > > + if (addr >= start && addr + len <= end)
>> >> > > + á á á á return true;
>> >> > > +
>> >> > > + return false;
>> >> > > +}
>> >> >
>> >> > Hmm, this check looks wrong to me: there's no guarantee
>> >> > that guest uses the first N entries in the table.
>> >> > E.g. it could use a single entry, but only the last one.
>> >>
>> >> Please check the PCI spec.
>> >
>> >
>> > This is pretty explicit in the spec: the the last paragraph in the below:
>> >
>> > IMPLEMENTATION NOTE
>> > Handling MSI-X Vector Shortages
>> >
>> > Handling MSI-X Vector Shortages
>> > For the case where fewer vectors are allocated to a function than desired,
>>
>> You may not notice the premise here.
>
> I noticed it.
>
>> Also check for "Table Size" would
>> help I think.
>
> It would help if msix_max_entries_nr was MSIX Table Size N
> (encoded in PCI config space as N - 1).
> Is this what it is? Maybe add this in some comment.
>
>> --
>> regards,
>> Yang, Sheng
>>
>> software-
>> > controlled aliasing as enabled by MSI-X is one approach for handling the situation. For
>> > example, if a function supports five queues, each with an associated MSI-X table entry, but
>> > only three vectors are allocated, the function could be designed for software still to configure
>> > all five table entries, assigning one or more vectors to multiple table entries. Software could
>> > assign the three vectors {A,B,C} to the five entries as ABCCC, ABBCC, ABCBA, or other
>> > similar combinations.
>> >
>> >
>> > Alternatively, the function could be designed for software to configure it (using a device-
>> > specific mechanism) to use only three queues and three MSI-X table entries. Software could
>> > assign the three vectors {A,B,C} to the five entries as ABC--, A-B-C, A--CB, or other similar
>> > combinations.
>> >
>> >
>> >
>> >> --
>> >> regards
>> >> Yang, Sheng
>> >>
>> >>
>> >> > > 2010/11/30 Yang, Sheng <sheng.yang@xxxxxxxxx>:
>> >> > > > On Tuesday 30 November 2010 17:10:11 lidong chen wrote:
>> >> > > >> sr-iov also meet this problem, MSIX mask waste a lot of cpu resource.
>> >> > > >>
>> >> > > >> I test kvm with sriov, which the vf driver could not disable msix.
>> >> > > >> so the host os waste a lot of cpu. ácpu rate of host os is 90%.
>> >> > > >>
>> >> > > >> then I test xen with sriov, there ara also a lot of vm exits caused by
>> >> > > >> MSIX mask.
>> >> > > >> but the cpu rate of xen and domain0 is less than kvm. cpu rate of xen
>> >> > > >> and domain0 is 60%.
>> >> > > >>
>> >> > > >> without sr-iov, the cpu rate of xen and domain0 is higher than kvm.
>> >> > > >>
>> >> > > >> so i think the problem is kvm waste more cpu resource to deal with
>> >> > > >> MSIX mask. and we can see how xen deal with MSIX mask.
>> >> > > >>
>> >> > > >> if this problem sloved, maybe with MSIX enabled, the performace is
>> >> > > >> better.
>> >> > > >
>> >> > > > Please refer to my posted patches for this issue.
>> >> > > >
>> >> > > > http://www.spinics.net/lists/kvm/msg44992.html
>> >> > > >
>> >> > > > --
>> >> > > > regards
>> >> > > > Yang, Sheng
>> >> > > >
>> >> > > >> 2010/11/23 Avi Kivity <avi@xxxxxxxxxx>:
>> >> > > >> > On 11/23/2010 09:27 AM, lidong chen wrote:
>> >> > > >> >> can you tell me something about this problem.
>> >> > > >> >> thanks.
>> >> > > >> >
>> >> > > >> > Which problem?
>> >> > > >> >
>> >> > > >> > --
>> >> > > >> > I have a truly marvellous patch that fixes the bug which this
>> >> > > >> > signature is too narrow to contain.
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe kvm" in
>> > the body of a message to majordomo@xxxxxxxxxxxxxxx
>> > More majordomo info at áhttp://vger.kernel.org/majordomo-info.html
>> >
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
