Commit-ID: 3729db5ca2b2000c660e5a5d0eb68b1053212cab Gitweb: http://git.kernel.org/tip/3729db5ca2b2000c660e5a5d0eb68b1053212cab Author: Andy Lutomirski <luto@xxxxxxx> AuthorDate: Mon, 23 May 2011 09:31:26 -0400 Committer: Thomas Gleixner <tglx@xxxxxxxxxxxxx> CommitDate: Tue, 24 May 2011 14:51:28 +0200 x86-64: Don't generate cmov in vread_tsc vread_tsc checks whether rdtsc returns something less than cycle_last, which is an extremely predictable branch. GCC likes to generate a cmov anyway, which is several cycles slower than a predicted branch. This saves a couple of nanoseconds. Signed-off-by: Andy Lutomirski <luto@xxxxxxx> Cc: Andi Kleen <andi@xxxxxxxxxxxxxx> Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> Cc: "David S. Miller" <davem@xxxxxxxxxxxxx> Cc: Eric Dumazet <eric.dumazet@xxxxxxxxx> Cc: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx> Cc: Borislav Petkov <bp@xxxxxxxxx> Link: http://lkml.kernel.org/r/%3C561280649519de41352fcb620684dfb22bad6bac.1306156808.git.luto%40mit.edu%3E Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx> --- arch/x86/kernel/tsc.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index 1e62442..24249a5 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -767,6 +767,7 @@ static cycle_t read_tsc(struct clocksource *cs) static cycle_t __vsyscall_fn vread_tsc(void) { cycle_t ret; + u64 last; /* * Empirically, a fence (of type that depends on the CPU) @@ -778,8 +779,21 @@ static cycle_t __vsyscall_fn vread_tsc(void) rdtsc_barrier(); ret = (cycle_t)vget_cycles(); - return ret >= VVAR(vsyscall_gtod_data).clock.cycle_last ? - ret : VVAR(vsyscall_gtod_data).clock.cycle_last; + last = VVAR(vsyscall_gtod_data).clock.cycle_last; + + if (likely(ret >= last)) + return ret; + + /* + * GCC likes to generate cmov here, but this branch is extremely + * predictable (it's just a funciton of time and the likely is + * very likely) and there's a data dependence, so force GCC + * to generate a branch instead. I don't barrier() because + * we don't actually need a barrier, and if this function + * ever gets inlined it will generate worse code. + */ + asm volatile (""); + return last; } #endif -- To unsubscribe from this list: send the line "unsubscribe linux-tip-commits" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html