Heavy clock drifts inside of KVM guests (qemu-kvm 2.1.2)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Dear list(s),

we are running a cluster of virtual machines on ganeti 2.15.2 on top of qemu-kvm 2.1.2. Hosts are Debian jessie (Kernel 3.16). We have big trouble with timekeeping on guests, no matter if these guests are lenny, squeeze, wheezy or jessie. Clock drifts heavily on some, others are fine. There is no obvious pattern on which guest this happens and on which not, but it seems the clock drifts heavier on guests that are idle and have nothing to do. Some more facts:

* Clocksource on all hosts are set to tsc
* Clocksource on all guests are set to kvm-clock
* CPUs on the hosts are Intel Xeons, all have these related flags: tsc, constant_tsc, nonstop_tsc. One host has tsc_deadline_timer as an extra, because it's a newer Xeon, but it doesn't seem to make a difference in drifting.
* Most guests complain about "Clocksource tsc unstable" in dmesg after boot (even when set to clocksource=kvm-clock)
* The hosts don't drift at all, we only see the issue in guests
* We don't use CPU pinning for KVM guests
* We run OpenNTPD 5.7p4 on both the hosts and the guests (we build this for ourselves, so the version is the same in every guest across every Debian version)

We tried different things to debug the cause of this issue but we're out if ideas now. This is what we have tested so far:

* Newer kernel from jessie-backports
* Newer qemu-kvm from jessie-backports
* Set clocksource=hpet on the hosts, this made drifts even worse (not tested yet: acpi_pm)
* Tried to run guests without kvm_steal_time setting
* Tried without running OpenNTPD on the guests

We have some hints along our investigations. Novell for example says that CPU pinning may help on drifting guests:
https://www.novell.com/support/kb/doc.php?id=7008698

Others suggested using an "failover clocksource" on the guests:
https://blog.laimbock.com/tag/clocksource_failover/

However, before we just continue to poke around in the dark, we wanted to hear input from virtualization experts that may have stumbled across something like this before. Any input would be greatly appreciated!

Please reply directly, as I'm not subscribed to the list. And sorry for cross-posting this to several lists at once, we just hope to get as much feedback as possible.

Thanks in advance,
Frank
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[Index of Archives]     [KVM Development]     [Libvirt Development]     [Libvirt Users]     [CentOS Virtualization]     [Netdev]     [Ethernet Bridging]     [Linux Wireless]     [Kernel Newbies]     [Security]     [Linux for Hams]     [Netfilter]     [Bugtraq]     [Yosemite Forum]     [MIPS Linux]     [ARM Linux]     [Linux RAID]     [Linux Admin]     [Samba]

  Powered by Linux