Re: Strange CPU usage pattern in SMP guest

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sun, Mar 21, 2010 at 05:17:38PM +0200, Avi Kivity wrote:
> On 03/21/2010 04:55 PM, Sebastian Hetze wrote:
>> On Sun, Mar 21, 2010 at 02:19:40PM +0200, Avi Kivity wrote:
>>    
>>> On 03/21/2010 02:02 PM, Sebastian Hetze wrote:
>>>      
>>>> 12:46:02     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
>>>> 12:46:03     all    0,20   11,35   10,96    8,96    0,40    2,99    0,00    0,00   65,14
>>>> 12:46:03       0    1,00   11,00    7,00   15,00    0,00    1,00    0,00    0,00   65,00
>>>> 12:46:03       1    0,00    7,14    2,04    6,12    1,02   11,22    0,00    0,00   72,45
>>>> 12:46:03       2    0,00   15,00    1,00   12,00    0,00    1,00    0,00    0,00   71,00
>>>> 12:46:03       3    0,00   11,00   23,00    8,00    0,00    0,00    0,00    0,00   58,00
>>>> 12:46:03       4    0,00    0,00   50,00    0,00    0,00    0,00    0,00    0,00   50,00
>>>> 12:46:03       5    0,00   13,00   20,00    4,00    0,00    1,00    0,00    0,00   62,00
>>>>
>>>> So it is only CPU4 that is showing this strange behaviour.
>>>>
>>>>        
>>> Can you adjust irqtop to only count cpu4?  or even just post a few 'cat
>>> /proc/interrupts' from that guest.
>>>
>>> Most likely the timer interrupt for cpu4 died.
>>>      
>> I've added two keys +/- to your irqtop to focus up and down
>> in the row of available CPUs.
>> The irqtop for CPU4 shows a constant number of 6 local timer interrupts
>> per update, while the other CPUs show various higher values:
>>
>> irqtop for cpu 4
>>
>>   eth0                                      188
>>   Rescheduling interrupts                   162
>>   Local timer interrupts                      6
>>   ata_piix                                    3
>>   TLB shootdowns                              1
>>   Spurious interrupts                         0
>>   Machine check exceptions                    0
>>
>>
>> irqtop for cpu 5
>>
>>   eth0                                      257
>>   Local timer interrupts                    251
>>   Rescheduling interrupts                   237
>>   Spurious interrupts                         0
>>   Machine check exceptions                    0
>>
>> So the timer interrupt for cpu4 is not completely dead but somehow
>> broken.
>
> That is incredibly weird.
>
>> What can cause this problem? Any way to speed it up again?
>>    
>
> The host has 8 cpus and is only running this 6 vcpu guest, yes?

The host is an dual quad core E5520 with hyperthrading enabled, so we
see 2x4x2=16 CPUs on the host. The guest is started with 6 CPUs.

> Can you confirm the other vcpus are ticking at 250 Hz?

The irqtop shows different numbers for local timer interrupts on the
other CPUs. The total number (summed up over all CPUs) varies between
something like 700 and 1400. Any CPU can be down to 10 and next update
up to 260. Only CPU4 stays at the 6 local timer interrupts.

>
> What does 'top' show running on cpu 4?  Pressing 'f' 'j' will add a  
> last-used-cpu field in the display.

The processes are not bound to a particular CPU, so the picture varies.
Here are two shots:

take1:

   15 root      RT  -5     0    0    0 S    0  0.0   0:01.70 4 migration/4
   16 root      15  -5     0    0    0 S    0  0.0   0:00.08 4 ksoftirqd/4
   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 4 watchdog/4
   25 root      15  -5     0    0    0 S    0  0.0   0:00.01 4 events/4
   35 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kintegrityd/4
   41 root      15  -5     0    0    0 S    0  0.0   0:00.03 4 kblockd/4
   50 root      15  -5     0    0    0 S    0  0.0   0:00.90 4 ata/4
   55 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kseriod
   66 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 aio/4
   73 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 crypto/4
   80 root      15  -5     0    0    0 S    0  0.0   2:11.71 4 scsi_eh_1
   87 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kmpathd/4
   95 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kondemand/4
  101 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kconservative/4
  103 root      10 -10     0    0    0 S    0  0.0   0:00.00 4 krfcommd
  681 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  686 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  691 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  737 root      15  -5     0    0    0 S    0  0.0   0:00.71 4 kjournald
  826 root      16  -4  2100  452  312 S    0  0.0   0:00.14 4 udevd
 1350 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kpsmoused
 1444 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kgameportd
 1718 root      15  -5     0    0    0 S    0  0.0   0:14.62 4 kjournald
 2108 statd     20   0  2252 1152  760 S    0  0.0   0:02.66 4 rpc.statd
 2117 root      15  -5     0    0    0 S    0  0.0   0:00.36 4 rpciod/4
 2123 root      15  -5     0    0    0 S    0  0.0   0:06.61 4 nfsiod
 2259 root      20   0  1696  444  440 S    0  0.0   0:00.00 4 getty
 2265 root      20   0  1696  444  440 S    0  0.0   0:00.00 4 getty
 2584 root      20   0  1860  656  604 S    0  0.0   0:01.92 4 automount
 2712 mysql     20   0  146m 4580 3464 S    0  0.1   0:59.71 4 mysqld
 2999 root      20   0 28944  12m 1704 S    0  0.2   1:23.04 4 console-kit-dae
 3834 root      20   0 13476 2640 1572 S    0  0.0   0:00.03 4 smbd
 4015 root      15  -5     0    0    0 S    0  0.0   7:22.33 4 nfsd
 4017 root      15  -5     0    0    0 S    0  0.0  10:46.87 4 nfsd
 4030 root      20   0  2644 1180  876 S    0  0.0   0:04.32 4 rpc.mountd
 4134 root      20   0  6164 1672 1344 S    0  0.0   0:00.37 4 master
 4311 root      20   0  5068 1796 1580 S    0  0.0   1:25.80 4 hald-addon-stor
 4560 root      20   0  2092  728  572 S    0  0.0   0:01.78 4 cron
 5053 www-data  20   0 33312 4092 1004 S    0  0.0   0:00.03 4 apache2

take 2:

   15 root      RT  -5     0    0    0 S    0  0.0   0:01.70 4 migration/4
   16 root      15  -5     0    0    0 S    0  0.0   0:00.08 4 ksoftirqd/4
   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 4 watchdog/4
   25 root      15  -5     0    0    0 S    0  0.0   0:00.01 4 events/4
   35 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kintegrityd/4
   41 root      15  -5     0    0    0 S    0  0.0   0:00.03 4 kblockd/4
   50 root      15  -5     0    0    0 S    0  0.0   0:00.90 4 ata/4
   55 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kseriod
   66 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 aio/4
   73 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 crypto/4
   87 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kmpathd/4
   95 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kondemand/4
  101 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kconservative/4
  103 root      10 -10     0    0    0 S    0  0.0   0:00.00 4 krfcommd
  681 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  686 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  691 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kdmflush
  737 root      15  -5     0    0    0 S    0  0.0   0:00.71 4 kjournald
  826 root      16  -4  2100  452  312 S    0  0.0   0:00.14 4 udevd
 1350 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kpsmoused
 1444 root      15  -5     0    0    0 S    0  0.0   0:00.00 4 kgameportd
 2091 daemon    20   0  1820  552  464 S    0  0.0   0:44.31 4 portmap
 2108 statd     20   0  2252 1152  760 S    0  0.0   0:02.66 4 rpc.statd
 2117 root      15  -5     0    0    0 S    0  0.0   0:00.36 4 rpciod/4
 2123 root      15  -5     0    0    0 S    0  0.0   0:06.61 4 nfsiod
 2259 root      20   0  1696  444  440 S    0  0.0   0:00.00 4 getty
 2265 root      20   0  1696  444  440 S    0  0.0   0:00.00 4 getty
 2712 mysql     20   0  146m 4580 3464 S    0  0.1   0:59.72 4 mysqld
 3834 root      20   0 13476 2640 1572 S    0  0.0   0:00.03 4 smbd
 4017 root      15  -5     0    0    0 S    0  0.0  10:46.87 4 nfsd
 4134 root      20   0  6164 1672 1344 S    0  0.0   0:00.37 4 master
 4311 root      20   0  5068 1796 1580 S    0  0.0   1:25.80 4 hald-addon-stor
 4560 root      20   0  2092  728  572 S    0  0.0   0:01.78 4 cron
 5053 www-data  20   0 33312 4092 1004 S    0  0.0   0:00.03 4 apache2

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux