On 03/22/2011 10:59 AM, Ben Nagy wrote:
On Tue, Mar 22, 2011 at 12:54 PM, Eric Dumazet<eric.dumazet@xxxxxxxxx> wrote: > Ben Nagy reported a scalability problem with KVM/QEMU that hit very hard > a single spinlock (idr_lock) in posix-timers code, on its 48 core > machine. Hi all, Thanks a lot for all the help so far. We've tested with Eric's patch. First up, here's our version of the patch for the current ubuntu kernel from git: http://paste.ubuntu.com/583668/ Here's top with 96 idle guests running: op - 16:47:53 up 1:09, 3 users, load average: 0.00, 0.01, 0.05 Tasks: 499 total, 3 running, 496 sleeping, 0 stopped, 0 zombie Cpu(s): 1.9%us, 3.2%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 99068656k total, 13121096k used, 85947560k free, 22192k buffers Swap: 2438140k total, 0k used, 2438140k free, 3597860k cached (much better!) Start of perf top: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- PerfTop: 10318 irqs/sec kernel:97.4% exact: 0.0% [1000Hz cycles], (all, 48 CPUs) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- samples pcnt function DSO _______ _____ ___________________________ ___________________________________________________________ 95444.00 59.3% __ticket_spin_lock [kernel.kallsyms] 12937.00 8.0% native_safe_halt [kernel.kallsyms] 6149.00 3.8% kvm_get_cs_db_l_bits /lib/modules/2.6.38-7-server/kernel/arch/x86/kvm/kvm.ko 5105.00 3.2% tg_load_down [kernel.kallsyms] 5088.00 3.2% svm_vcpu_run /lib/modules/2.6.38-7-server/kernel/arch/x86/kvm/kvm-amd.ko 4807.00 3.0% kvm_set_pfn_dirty /lib/modules/2.6.38-7-server/kernel/arch/x86/kvm/kvm.ko 2855.00 1.8% ktime_get [kernel.kallsyms] 1535.00 1.0% find_busiest_group [kernel.kallsyms] 1386.00 0.9% find_next_bit [kernel.kallsyms] Start of perf report -g 55.26% kvm [kernel.kallsyms] [k] __ticket_spin_lock | --- __ticket_spin_lock | |--94.68%-- _raw_spin_lock | | | |--97.55%-- double_rq_lock | | load_balance | | idle_balance | | schedule | | | | | |--60.56%-- schedule_hrtimeout_range_clock | | | schedule_hrtimeout_range | | | poll_schedule_timeout | | | do_select | | | core_sys_select | | | sys_select | | | system_call_fastpath
Looks like the posix-timer issue is completely gone, to be replaced by the load balancer.
Copying peterz. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
