From: Jeremy Fitzhardinge <jeremy.fitzhardinge@xxxxxxxxxx>
[ Changes since last posting:
- fix bugs exposed by the cold light of testing
- make the "slow flag" read in unlock cover the whole lock
to force ordering WRT the unlock write
- when kicking on unlock, only look for the CPU *we* released
(ie, head value the unlock resulted in), rather than re-reading
the new head and kicking on that basis
- enable PV ticketlocks in Xen HVM guests
]
NOTE: this series is available in:
git://github.com/jsgf/linux-xen.git upstream/pvticketlock-slowflag
and is based on the previously posted ticketlock cleanup series in
git://github.com/jsgf/linux-xen.git upstream/ticketlock-cleanup
This series replaces the existing paravirtualized spinlock mechanism
with a paravirtualized ticketlock mechanism.
Ticket locks have an inherent problem in a virtualized case, because
the vCPUs are scheduled rather than running concurrently (ignoring
gang scheduled vCPUs). This can result in catastrophic performance
collapses when the vCPU scheduler doesn't schedule the correct "next"
vCPU, and ends up scheduling a vCPU which burns its entire timeslice
spinning. (Note that this is not the same problem as lock-holder
preemption, which this series also addresses; that's also a problem,
but not catastrophic).
(See Thomas Friebel's talk "Prevent Guests from Spinning Around"
http://www.xen.org/files/xensummitboston08/LHP.pdf for more details.)
Currently we deal with this by having PV spinlocks, which adds a layer
of indirection in front of all the spinlock functions, and defining a
completely new implementation for Xen (and for other pvops users, but
there are none at present).
PV ticketlocks keeps the existing ticketlock implemenentation
(fastpath) as-is, but adds a couple of pvops for the slow paths:
- If a CPU has been waiting for a spinlock for SPIN_THRESHOLD
iterations, then call out to the __ticket_lock_spinning() pvop,
which allows a backend to block the vCPU rather than spinning. This
pvop can set the lock into "slowpath state".
- When releasing a lock, if it is in "slowpath state", the call
__ticket_unlock_kick() to kick the next vCPU in line awake. If the
lock is no longer in contention, it also clears the slowpath flag.
The "slowpath state" is stored in the LSB of the within the lock
ticket. This has the effect of reducing the max number of CPUs by
half (so, a "small ticket" can deal with 128 CPUs, and "large ticket"
32768).
This series provides a Xen implementation, but it should be
straightforward to add a KVM implementation as well.
Overall, it results in a large reduction in code, it makes the native
and virtualized cases closer, and it removes a layer of indirection
around all the spinlock functions.
The fast path (taking an uncontended lock which isn't in "slowpath"
state) is optimal, identical to the non-paravirtualized case.
The inner part of ticket lock code becomes:
inc = xadd(&lock->tickets, inc);
inc.tail &= ~TICKET_SLOWPATH_FLAG;
if (likely(inc.head == inc.tail))
goto out;
for (;;) {
unsigned count = SPIN_THRESHOLD;
do {
if (ACCESS_ONCE(lock->tickets.head) == inc.tail)
goto out;
cpu_relax();
} while (--count);
__ticket_lock_spinning(lock, inc.tail);
}
out: barrier();
which results in:
push %rbp
mov %rsp,%rbp
mov $0x200,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f
pop %rbp
retq
### SLOWPATH START
1: and $-2,%edx
movzbl %dl,%esi
2: mov $0x800,%eax
jmp 4f
3: pause
sub $0x1,%eax
je 5f
4: movzbl (%rdi),%ecx
cmp %cl,%dl
jne 3b
pop %rbp
retq
5: callq *__ticket_lock_spinning
jmp 2b
### SLOWPATH END
with CONFIG_PARAVIRT_SPINLOCKS=n, the code has changed slightly, where
the fastpath case is straight through (taking the lock without
contention), and the spin loop is out of line:
push %rbp
mov %rsp,%rbp
mov $0x100,%eax
lock xadd %ax,(%rdi)
movzbl %ah,%edx
cmp %al,%dl
jne 1f
pop %rbp
retq
### SLOWPATH START
1: pause
movzbl (%rdi),%eax
cmp %dl,%al
jne 1b
pop %rbp
retq
### SLOWPATH END
The unlock code is very straightforward:
prev = *lock;
__ticket_unlock_release(lock);
if (unlikely(__ticket_in_slowpath(lock)))
__ticket_unlock_slowpath(lock, prev);
which generates:
push %rbp
mov %rsp,%rbp
movzwl (%rdi),%esi
addb $0x2,(%rdi)
movzwl (%rdi),%eax
testb $0x1,%ah
jne 1f
pop %rbp
retq
### SLOWPATH START
1: movzwl (%rdi),%edx
movzbl %dh,%ecx
mov %edx,%eax
and $-2,%ecx # clear TICKET_SLOWPATH_FLAG
mov %cl,%dh
cmp %dl,%cl # test to see if lock is uncontended
je 3f
2: movzbl %dl,%esi
callq *__ticket_unlock_kick # kick anyone waiting
pop %rbp
retq
3: lock cmpxchg %dx,(%rdi) # use cmpxchg to safely write back flag
jmp 2b
### SLOWPATH END
The fastpath is pretty straightforward, but it is definitely more
complex than a simple "addb $1,(%rdi)" - which is still generated (and
inlined) when PARAVIRT_SPINLOCKS is disabled.
Thoughts? Comments? Suggestions?
Thanks,
J
Jeremy Fitzhardinge (9):
x86/spinlocks: replace pv spinlocks with pv ticketlocks
x86/ticketlock: don't inline _spin_unlock when using paravirt
spinlocks
x86/ticketlock: collapse a layer of functions
xen/pvticketlock: Xen implementation for PV ticket locks
x86/pvticketlock: use callee-save for lock_spinning
x86/ticketlocks: when paravirtualizing ticket locks, increment by 2
x86/ticketlock: add slowpath logic
xen/pvticketlock: allow interrupts to be enabled while blocking
Stefano Stabellini (1):
xen: enable PV ticketlocks on HVM Xen
arch/x86/Kconfig | 3 +
arch/x86/include/asm/paravirt.h | 30 +---
arch/x86/include/asm/paravirt_types.h | 10 +-
arch/x86/include/asm/spinlock.h | 160 ++++++++++++-----
arch/x86/include/asm/spinlock_types.h | 16 ++-
arch/x86/kernel/paravirt-spinlocks.c | 16 +--
arch/x86/xen/smp.c | 1 +
arch/x86/xen/spinlock.c | 315 ++++++++------------------------
kernel/Kconfig.locks | 2 +-
9 files changed, 217 insertions(+), 336 deletions(-)
--
1.7.6
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
