On 04/16/2013 07:49 AM, Stefan Hajnoczi wrote: Hi Stefan, > Besides the kvm_stat, general performance data from the host is useful > when dealing with high load averages. > > Do you have vmstat or sar data for periods of time when the machine was > slow? > > Stefan We do have a rather exhaustive log on the guest. As for the host, we did not find anything suspicious except for the kvm_stat output. So we did not log any more than that. Here is the output of "vmstat 5 5" on the guest: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 84 0 19596 104404 60 21932616 0 0 232 110 9 2 7 2 90 1 80 0 19596 98100 60 21933920 0 0 106 119 854 912 79 21 0 0 89 0 19596 94216 60 21932764 0 0 106 223 864 886 79 21 0 0 87 0 19596 95848 60 21927612 0 0 82 47 856 906 79 21 0 0 Load average at that time: 75 (1:20 AM) The guest seems to have a hard time scheduling tasks. The log output, which is triggered by a simple cronjob that executes commands like "vmstat", "atop" or simply appending some information from /proc and /sys to the log (in a sequential manner) is a bit scrambled (i.e. the expected order of the output is not kept, most likely because the cronjobs get piled up). This can also be seen on the rather large numbers in the first column (there is no workload scheduled for that time and virtually no one is using the system, the whole thing just seems to happen out of thin air). The huge number of runnable tasks fits the high load average, but we could not see the reason why the tasks are piling up. There is/was no apparent I/O issue on disk or network and no messages from the kernel at that time. Also, there was no swap activity on either guest or host (host has swap disabled). For comparison, here is the output from 22:55 (10:55 PM), a couple of hours before the output from above: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 36 97848 60 24259656 0 0 227 111 7 1 7 2 91 1 6 0 36 121532 60 24220340 0 0 106 327 974 539 31 22 46 0 9 0 36 126108 60 24212756 0 0 6 0 1019 517 19 7 73 0 2 0 36 125264 60 24212780 0 0 0 26 1007 481 25 6 69 0 6 0 36 147564 60 24212808 0 0 4 120 1345 1161 25 10 65 0 If you need any more info, just let me know. Best regards, Martin -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
