Avi Kivity wrote: > (copying lkml and some scheduler folk) > > On 04/10/2010 11:16 AM, Zhang, Xiantao wrote: >> Hi, all >> We are working on the scalability work for KVM guests, and found >> one big issue exists in linux scheduler and it may impact guest's >> performance and scalability a lot for some special workloads running >> in VM. In the current Linux scheduler, there are some features to >> enhance App's performance which are defined in the file >> kvm.git/kernel/sched_features.h. Certainly, they are mostly >> beneficial optimizations to improve system's performance, but >> unluckily, some of them may hurt VM's performance and scalablity in >> KVM case We know that if two or more vcpus of one guests are >> scheduled to one same logical processor, same CPU utilization may >> generate less valid output due mutual lock in VM's OS than that are >> scheduled to different logical processors .And we also know that >> VM's vcpus are emulated or executed through the threads of Qemu for >> KVM. If the vcpu threads of qemu are often pulled to one same >> logical processor by some features of Linux scheduler, kvm >> guests'performance may be hurt a lot. In our performance testing, >> the results also show this performance bottleneck due to this issue. > > What was the performance hit? What was your I/O setup (image format, > using aio?) The issue only happens when vcpu number is over-committed(e.g. vcpu/pcpu>2) and physical cpus are saturated. For example, when run webbench in windows OS in this case, its performance drops by 80%. In our experiment, we are using image file through virtio, and I think aio should be used by default also. >> After analysis about Linux scheduler, we found it is indeed caused >> by the known features of Linux schduler, such as AFFINE_WAKEUPS, >> SYNC_WAKEUPS etc. With these features on, linux schduler often tries >> to schedule the vcpu threads of one guests to one same logical >> processor when vcpus are over-committed and logical processors are >> saturated. Once the vcpu threads of one VM are scheduled to the same >> LP, system performance drops dramatically with some workloads(like >> webbench running in windows OS). >> > > Were the affine wakeups due to the kernel (emulated guest IPIs) or > qemu? We have basic guesses about the reasone, one is wakeup affinity between vcpu threads due to IPI, and the other is wakeup affinity between io theads and vcpu threads. >> To verify this finding, we also worked out a simple patch >> attached in the mail to dynamially switch off the two sheduler >> features mentioned above when scheduler knows the scheduling tasks >> are vcpu threads, and we found the the whole system's performance >> and scalability are improved a lot. Certatinly, this patch is not >> good for upstream, but it can enlighten us to think how to optimize >> Linux scheduler and we also want to initiate the discussion about >> how to make LINUX's scheduler more friendly to virtualization. >> Besides, this issue maybe not only kvm's special issue, insteadly it >> should be a common issue for host-based VMs, and we also expect that >> we can have an elegant solution to thoroughly resolve the >> performance or scalability gap compared with hypervisor-based VMs. >> > > Most likely it also hits non-virtualized loads as well. If the > scheduler pulls two long-running threads to the same cpu, performance > will take a hit. Since the hit only happens when physical cpus are saturated, and sheduling non-virtualized multiple threads of one process to same processor can benefit the performance due to cache share or other affinities, but you know it hurts performance a lot once schedule two vcpu theads to a same processor due to mutual spin-lock in guests. Xiantao -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
