Re: [RFC 0/3] KVM: x86: introduce pv feature lazy tscdeadline

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On 2023.07.13 21:32, Xiaoyao Li wrote:
> On 7/13/2023 10:50 AM, Wang Jianchao wrote:
>>
>>
>> On 2023.07.13 02:14, Zhi Wang wrote:
>>> On Fri,  7 Jul 2023 14:17:58 +0800
>>> Wang Jianchao <jianchwa@xxxxxxxxxxx> wrote:
>>>
>>>> Hi
>>>>
>>>> This patchset attemps to introduce a new pv feature, lazy tscdeadline.
>>>> Everytime guest write msr of MSR_IA32_TSC_DEADLINE, a vm-exit occurs
>>>> and host side handle it. However, a lot of the vm-exit is unnecessary
>>>> because the timer is often over-written before it expires.
>>>>
>>>> v : write to msr of tsc deadline
>>>> | : timer armed by tsc deadline
>>>>
>>>>           v v v v v        | | | | |
>>>> --------------------------------------->  Time
>>>>
>>>> The timer armed by msr write is over-written before expires and the
>>>> vm-exit caused by it are wasted. The lazy tscdeadline works as following,
>>>>
>>>>           v v v v v        |       |
>>>> --------------------------------------->  Time
>>>>                            '- arm -'
>>>>
>>>
>>> Interesting patch.
>>>
>>> I am a little bit confused of the chart above. It seems the write of MSR,
>>> which is said to cause VM exit, is not reduced in the chart of lazy
>>> tscdeadline, only the times of arm are getting less. And the benefit of
>>> lazy tscdeadline is said coming from "less vm exit". Maybe it is better
>>> to imporve the chart a little bit to help people jump into the idea
>>> easily?
>>
>> Thanks so much for you comment and sorry for my poor chart.
>>
>> Let me try to rework the chart.
>>
>> Before this patch, every time guest start or modify a hrtimer, we need to write the msr of tsc deadline,
>> a vm-exit occurs and host arms a hv or sw timer for it.
>>
>>
>> w: write msr
>> x: vm-exit
>> t: hv or sw timer
>>
>>
>> Guest
>>           w
>> --------------------------------------->  Time
>> Host     x              t
>>  
>> However, in some workload that needs setup timer frequently, msr of tscdeadline is usually overwritten
>> many times before the timer expires. And every time we modify the tscdeadline, a vm-exit ocurrs
>>
>>
>> 1. write to msr with t0
>>
>> Guest
>>           w0
>> ---------------------------------------->  Time
>> Host     x0             t0
>>
>>   2. write to msr with t1
>> Guest
>>               w1
>> ------------------------------------------>  Time
>> Host         x1          t0->t1
>>
>>
>> 2. write to msr with t2
>> Guest
>>                  w2
>> ------------------------------------------>  Time
>> Host            x2          t1->t2
>>  
>> 3. write to msr with t3
>> Guest
>>                      w3
>> ------------------------------------------>  Time
>> Host                x3           t2->t3
>>
>>
>>
>> What this patch want to do is to eliminate the vm-exit of x1 x2 and x3 as following,
>>
>>
>> Firstly, we have two fields shared between guest and host as other pv features, saying,
>>   - armed, the value of tscdeadline that has a timer in host side, only updated by __host__ side
>>   - pending, the next value of tscdeadline, only updated by __guest__ side
>>
>>
>> 1. write to msr with t0
>>
>>               armed   : t0
>>               pending : t0
>> Guest
>>           w0
>> ---------------------------------------->  Time
>> Host     x0             t0
>>
>> vm-exit occurs and arms a timer for t0 in host side
> 
> What's the initial value of @armed and @pending?

Both of them are zero.

@armed is only updated by host
@pending is updated by guest

Guest side will check @armed, it it is zero, jumps to wrmsrl

> 
>>   2. write to msr with t1
>>
>>               armed   : t0
>>               pending : t1
>>
>> Guest
>>               w1
>> ------------------------------------------>  Time
>> Host                     t0
>>
>> the value of tsc deadline that has been armed, namely t0, is smaller than t1, needn't to write
>> to msr but just update pending
> 
> if t1 < t0, then it triggers the vm exit, right?

Yes. If new tsc deadline value is smaller than @armed, namely t1 here, it jumps to wrmsrl

> And in this case, I think @armed will be updated to t1. What about pending? will it get updated to t1 or not?

Yes, the guest jumps to wrmsrl and causes a vm-exit, the host side will update the @armed and re-arm the timer


Thanks
Jianchao

> 
>>
>> 3. write to msr with t2
>>
>>               armed   : t0
>>               pending : t2
>>   Guest
>>                  w2
>> ------------------------------------------>  Time
>> Host                      t0
>>   Similar with step 2, just update pending field with t2, no vm-exit
>>
>>
>> 4.  write to msr with t3
>>
>>               armed   : t0
>>               pending : t3
>>
>> Guest
>>                      w3
>> ------------------------------------------>  Time
>> Host                       t0
>> Similar with step 2, just update pending field with t3, no vm-exit
>>
>>
>> 5.  t0 expires, arm t3
>>
>>               armed   : t3
>>               pending : t3
>>
>>
>> Guest
>>                              ------------------------------------------>  Time
>> Host                       t0  ------> t3
>>
>> t0 is fired, it checks the pending field and re-arm a timer based on it.
>>
>>
>> Here is the core ideal of this patch ;)
>>
>>
>> Thanks
>> Jianchao
>>
>>>
>>>> The 1st timer is responsible for arming the next timer. When the armed
>>>> timer is expired, it will check pending and arm a new timer.
>>>>
>>>> In the netperf test with TCP_RR on loopback, this lazy_tscdeadline can
>>>> reduce vm-exit obviously.
>>>>
>>>>                           Close               Open
>>>> --------------------------------------------------------
>>>> VM-Exit
>>>>               sum         12617503            5815737
>>>>              intr      0% 37023            0% 33002
>>>>             cpuid      0% 1                0% 0
>>>>              halt     19% 2503932         47% 2780683
>>>>         msr-write     79% 10046340        51% 2966824
>>>>             pause      0% 90               0% 84
>>>>     ept-violation      0% 584              0% 336
>>>>     ept-misconfig      0% 0                0% 2
>>>> preemption-timer      0% 29518            0% 34800
>>>> -------------------------------------------------------
>>>> MSR-Write
>>>>              sum          10046455            2966864
>>>>          apic-icr     25% 2533498         93% 2781235
>>>>      tsc-deadline     74% 7512945          6% 185629
>>>>
>>>> This patchset is made and tested on 6.4.0, includes 3 patches,
>>>>
>>>> The 1st one adds necessary data structures for this feature
>>>> The 2nd one adds the specific msr operations between guest and host
>>>> The 3rd one are the one make this feature works.
>>>>
>>>> Any comment is welcome.
>>>>
>>>> Thanks
>>>> Jianchao
>>>>
>>>> Wang Jianchao (3)
>>>>     KVM: x86: add msr register and data structure for lazy tscdeadline
>>>>     KVM: x86: exchange info about lazy_tscdeadline with msr
>>>>     KVM: X86: add lazy tscdeadline support to reduce vm-exit of msr-write
>>>>
>>>>
>>>>   arch/x86/include/asm/kvm_host.h      |  10 ++++++++
>>>>   arch/x86/include/uapi/asm/kvm_para.h |   9 +++++++
>>>>   arch/x86/kernel/apic/apic.c          |  47 ++++++++++++++++++++++++++++++++++-
>>>>   arch/x86/kernel/kvm.c                |  13 ++++++++++
>>>>   arch/x86/kvm/cpuid.c                 |   1 +
>>>>   arch/x86/kvm/lapic.c                 | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
>>>>   arch/x86/kvm/lapic.h                 |   4 +++
>>>>   arch/x86/kvm/x86.c                   |  26 ++++++++++++++++++++
>>>>   8 files changed, 229 insertions(+), 9 deletions(-)
>>>
> 



[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux