On Mon, Jan 04, 2016 at 02:33:12PM -0800, Andy Lutomirski wrote:
> On Mon, Jan 4, 2016 at 12:26 PM, Marcelo Tosatti <mtosatti@xxxxxxxxxx> wrote:
> > On Sun, Dec 20, 2015 at 03:05:41AM -0800, Andy Lutomirski wrote:
> >> From: Andy Lutomirski <luto@xxxxxxxxxxxxxx>
> >>
> >> The pvclock vdso code was too abstracted to understand easily and
> >> excessively paranoid. Simplify it for a huge speedup.
> >>
> >> This opens the door for additional simplifications, as the vdso no
> >> longer accesses the pvti for any vcpu other than vcpu 0.
> >>
> >> Before, vclock_gettime using kvm-clock took about 45ns on my machine.
> >> With this change, it takes 29ns, which is almost as fast as the pure TSC
> >> implementation.
> >>
> >> Reviewed-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> >> Signed-off-by: Andy Lutomirski <luto@xxxxxxxxxxxxxx>
> >> ---
> >> arch/x86/entry/vdso/vclock_gettime.c | 81 ++++++++++++++++++++----------------
> >> 1 file changed, 46 insertions(+), 35 deletions(-)
> >>
> >> diff --git a/arch/x86/entry/vdso/vclock_gettime.c b/arch/x86/entry/vdso/vclock_gettime.c
> >> index ca94fa649251..c325ba1bdddf 100644
> >> --- a/arch/x86/entry/vdso/vclock_gettime.c
> >> +++ b/arch/x86/entry/vdso/vclock_gettime.c
> >> @@ -78,47 +78,58 @@ static notrace const struct pvclock_vsyscall_time_info *get_pvti(int cpu)
> >>
> >> static notrace cycle_t vread_pvclock(int *mode)
> >> {
> >> - const struct pvclock_vsyscall_time_info *pvti;
> >> + const struct pvclock_vcpu_time_info *pvti = &get_pvti(0)->pvti;
> >> cycle_t ret;
> >> - u64 last;
> >> - u32 version;
> >> - u8 flags;
> >> - unsigned cpu, cpu1;
> >> -
> >> + u64 tsc, pvti_tsc;
> >> + u64 last, delta, pvti_system_time;
> >> + u32 version, pvti_tsc_to_system_mul, pvti_tsc_shift;
> >>
> >> /*
> >> - * Note: hypervisor must guarantee that:
> >> - * 1. cpu ID number maps 1:1 to per-CPU pvclock time info.
> >> - * 2. that per-CPU pvclock time info is updated if the
> >> - * underlying CPU changes.
> >> - * 3. that version is increased whenever underlying CPU
> >> - * changes.
> >> + * Note: The kernel and hypervisor must guarantee that cpu ID
> >> + * number maps 1:1 to per-CPU pvclock time info.
> >> + *
> >> + * Because the hypervisor is entirely unaware of guest userspace
> >> + * preemption, it cannot guarantee that per-CPU pvclock time
> >> + * info is updated if the underlying CPU changes or that that
> >> + * version is increased whenever underlying CPU changes.
> >> *
> >> + * On KVM, we are guaranteed that pvti updates for any vCPU are
> >> + * atomic as seen by *all* vCPUs. This is an even stronger
> >> + * guarantee than we get with a normal seqlock.
> >> + *
> >> + * On Xen, we don't appear to have that guarantee, but Xen still
> >> + * supplies a valid seqlock using the version field.
> >> +
> >> + * We only do pvclock vdso timing at all if
> >> + * PVCLOCK_TSC_STABLE_BIT is set, and we interpret that bit to
> >> + * mean that all vCPUs have matching pvti and that the TSC is
> >> + * synced, so we can just look at vCPU 0's pvti.
> >> */
> >> - do {
> >> - cpu = __getcpu() & VGETCPU_CPU_MASK;
> >> - /* TODO: We can put vcpu id into higher bits of pvti.version.
> >> - * This will save a couple of cycles by getting rid of
> >> - * __getcpu() calls (Gleb).
> >> - */
> >> -
> >> - pvti = get_pvti(cpu);
> >> -
> >> - version = __pvclock_read_cycles(&pvti->pvti, &ret, &flags);
> >> -
> >> - /*
> >> - * Test we're still on the cpu as well as the version.
> >> - * We could have been migrated just after the first
> >> - * vgetcpu but before fetching the version, so we
> >> - * wouldn't notice a version change.
> >> - */
> >> - cpu1 = __getcpu() & VGETCPU_CPU_MASK;
> >> - } while (unlikely(cpu != cpu1 ||
> >> - (pvti->pvti.version & 1) ||
> >> - pvti->pvti.version != version));
> >> -
> >> - if (unlikely(!(flags & PVCLOCK_TSC_STABLE_BIT)))
> >> +
> >> + if (unlikely(!(pvti->flags & PVCLOCK_TSC_STABLE_BIT))) {
> >> *mode = VCLOCK_NONE;
> >> + return 0;
> >> + }
> >> +
> >> + do {
> >> + version = pvti->version;
> >> +
> >> + /* This is also a read barrier, so we'll read version first. */
> >> + tsc = rdtsc_ordered();
> >> +
> >> + pvti_tsc_to_system_mul = pvti->tsc_to_system_mul;
> >> + pvti_tsc_shift = pvti->tsc_shift;
> >> + pvti_system_time = pvti->system_time;
> >> + pvti_tsc = pvti->tsc_timestamp;
> >> +
> >> + /* Make sure that the version double-check is last. */
> >> + smp_rmb();
> >> + } while (unlikely((version & 1) || version != pvti->version));
> >
> > Andy,
> >
> > What happens if PVCLOCK_TSC_STABLE_BIT is disabled here?
>
> Do you mean what happens if it's disabled in the loop part after the
> first check? If that's actually possible, I'll do a follow-up to bail
> if that happens by moving the check into the loop.
>
> --Andy
It is possible.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
