On Mon, 2018-01-22 at 20:27 +0800, Wanpeng Li wrote: > 2018-01-22 20:08 GMT+08:00 Mike Galbraith <efault@xxxxxx>: > > On Mon, 2018-01-22 at 19:47 +0800, Wanpeng Li wrote: > >> Hi all, > >> > >> We can observe unixbench context switch performance is heavily > >> influenced by cpu topology which is exposed to the guest. the score is > >> posted below, bigger is better, both the guest and the host kernel are > >> 3.15-rc3(we can also reproduce against centos 7.4 693 guest/host), LLC > >> is exposed to the guest, kvm adaptive halt-polling is default enabled, > >> then start a guest w/ 8 logical cpus. > >> > >> > >> > >> unixbench context switch > >> -smp 8, sockets=8, cores=1, threads=1 382036 > >> -smp 8, sockets=4, cores=2, threads=1 132480 > >> -smp 8, sockets=2, cores=4, threads=1 128032 > >> -smp 8, sockets=2, cores=2, threads=2 131767 > >> -smp 8, sockets=1, cores=4, threads=2 132742 > >> -smp 8, sockets=1, cores=4, threads=2 (guest w/ nohz=off idle=poll) 331471 > >> > >> I can observe there are a lot of reschedule IPIs sent from one vCPU to > >> another vCPU, the context switch workload switches between running and > >> idle frequently which results in HLT instruction in the idle path, I > >> use idle=poll to avoid vmexit due to HLT and to avoid reschedule IPIs > >> since idle task checks TIF_NEED_RESCHED flags in a loop, nohz=off can > >> stop to program lapic timer/other nohz stuffs. Any idea why sockets=8 > >> can get best performance? > > > > Probably because with that topology, there is no shared llc, thus no > > cross-core scheduling, micro-benchmark waker/wakee are stacked. If > > your benchmark does nothing but schedule, stacking makes beautiful (but > > utterly meaningless) numbers. > > The waker and wakee are just sporadic on the same logical cpu in the > guest(-smp 8, sockets=8, cores=1, threads=1) during the testing, in > addition, binding the waker/wakee to one logical cpu in the guest(-smp > 8, sockets=1, cores=4, threads=2) also can get the performance as > better as 8 sockets setup. Here, with tip.today and that topology, context1 does stack up on one core. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4218 root 20 0 4048 808 732 R 52.16 0.022 0:12.77 4 context1 4219 root 20 0 4048 80 0 S 47.18 0.002 0:11.96 4 context1 There's a bit of bouncing, but the two stack right back up. But whatever, what Peter said, the benchmark should pin itself to do this. -Mike
