Update: tried 5.16.11 kernel. Same behavior. > -----Original Message----- > From: Eugene Huang > Sent: Tuesday, March 1, 2022 11:04 AM > To: Marc Zyngier <maz@xxxxxxxxxx> > Cc: kvmarm@xxxxxxxxxxxxxxxxxxxxx > Subject: RE: Timer delays in VM > > Thanks Marc for the reply. Please see inline below marked with [EH]. > > > -----Original Message----- > > From: Marc Zyngier <maz@xxxxxxxxxx> > > Sent: Monday, February 28, 2022 1:03 PM > > To: Eugene Huang <eugeneh@xxxxxxxxxx> > > Cc: kvmarm@xxxxxxxxxxxxxxxxxxxxx > > Subject: Re: Timer delays in VM > > > > External email: Use caution opening links or attachments > > > > > > [Please don't send HTML email and stick to plain text] > > > > On 2022-02-28 18:02, Eugene Huang wrote: > > > Hi, > > > > > > I am running qemu on an arm64 CentOS host. Inside a ubuntu VM, a > > > > I assume that by this you mean QEMU as the VMM for a KVM guest, right? > > [EH] Yes. > > > > > > process runs a timer created using timer_t: > > > > > > ev.sigev_notify_function = m_callback; > > > > > > … > > > > > > timer_create(CLOCK_MONOTONIC, &ev, &m_timer_t); > > > > > > This timer sometimes has significant delays. For example, the 50 ms > > > timer can have a callback delay of 100ms. > > > > > > I did a host kernel trace and see a lot of WFx kvm_exits, and the > > > following events between kvm_exit and kvm_entry: > > > > > > kvm_exit > > > > > > kvm_wfx_arm64 > > > > > > kvm_get_timer_map > > > > > > sched_switch > > > > > > kvm_timer_save_state > > > > > > kvm_timer_update_irq > > > > > > vgic_update_irq_pending > > > > > > kvm_timer_restore_state > > > > > > kvm_vcpu_wakeup > > > > > > kvm_arm_setup_debug > > > > > > kvm_arm_set_dreg32 > > > > > > kvm_entry > > > > All of this is perfectly normal (guest hits WFI from its idle loop, no > > interrupt is pending, trap to EL2, schedule out, schedule back in, reenter > the guest). > > > > > > > > I have the following questions: > > > > > > * Why there are a lot WFx exits? Is the timer dependent on it? > > > > That's most probably because your vcpu goes idle and execute WFI to > > Wait For an Interrupt. As no interrupt is pending, the vcpu exits so > > that the host can do something useful until it gets an interrupt that is > targeted at the vcpu. > > On an idle VM, this probably happens 100s of times a second. > > > > > * Does this timer rely on kvm timer irq injection? > > > > Yes. A timer interrupt is always injected in SW. But the timer > > interrupt can either come from the HW timer itself (the VM was running > > while the timer expired), or from a SW timer that KVM as setup if the > > guest was blocked on WFI. > > <EH> Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is a HW > timer, correct? > There is an armvtimer implementation in QEMU 6.1+. Does this armvtimer > make a difference? > > > > > > * What can be any possible causes for the timer delay? Are > > > there some locking mechanisms which can cause the delay? > > > > This completely depend on how loaded your host is, the respective > > priorities of the various processes, and a million of other things. > > This is no different from the same userspace running on the host. > > It also depends on the *guest* kernel, by the way. > > <EH> Our guest kernel is 5.4. How is the *guest* kernel involved? Can you > give an example? Do you have suggestions on the guest kernel version as > well. > > > > > There are of course locks all over the place, but that's the very > > nature of the beast. > > > > > * What parameters can tune this timer? > > > > None. You may want to check whether the delay is observed when the VM > > has hit WFI or not. > > <EH> Yes, delay is observed after vm_exit because of WFx (not sure WFI or > WFE) but only when on a different vCPU in the same VM some workload is > started. Since we pin that workload to its own vCPU, in theory, it should not > affect the timing of another vCPU. > > > > > You also don't mention what host kernel version you are running. > > In general, please try and reproduce the issue using the latest kernel > > version > > (5.16 at the moment). Please also indicate what HW you are using. > > <EH> Tried 5.15 and 5.4 kernels. Both have the issue. Do you think 5.16 can > make a difference? The HW is an Ampere Altra system. > > > > > Thanks, > > > > M. > > -- > > Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
