On Sun, Mar 21, 2010 at 05:17:38PM +0200, Avi Kivity wrote: > On 03/21/2010 04:55 PM, Sebastian Hetze wrote: >> On Sun, Mar 21, 2010 at 02:19:40PM +0200, Avi Kivity wrote: >> >>> On 03/21/2010 02:02 PM, Sebastian Hetze wrote: >>> >>>> 12:46:02 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 12:46:03 all 0,20 11,35 10,96 8,96 0,40 2,99 0,00 0,00 65,14 >>>> 12:46:03 0 1,00 11,00 7,00 15,00 0,00 1,00 0,00 0,00 65,00 >>>> 12:46:03 1 0,00 7,14 2,04 6,12 1,02 11,22 0,00 0,00 72,45 >>>> 12:46:03 2 0,00 15,00 1,00 12,00 0,00 1,00 0,00 0,00 71,00 >>>> 12:46:03 3 0,00 11,00 23,00 8,00 0,00 0,00 0,00 0,00 58,00 >>>> 12:46:03 4 0,00 0,00 50,00 0,00 0,00 0,00 0,00 0,00 50,00 >>>> 12:46:03 5 0,00 13,00 20,00 4,00 0,00 1,00 0,00 0,00 62,00 >>>> >>>> So it is only CPU4 that is showing this strange behaviour. >>>> >>>> >>> Can you adjust irqtop to only count cpu4? or even just post a few 'cat >>> /proc/interrupts' from that guest. >>> >>> Most likely the timer interrupt for cpu4 died. >>> >> I've added two keys +/- to your irqtop to focus up and down >> in the row of available CPUs. >> The irqtop for CPU4 shows a constant number of 6 local timer interrupts >> per update, while the other CPUs show various higher values: >> >> irqtop for cpu 4 >> >> eth0 188 >> Rescheduling interrupts 162 >> Local timer interrupts 6 >> ata_piix 3 >> TLB shootdowns 1 >> Spurious interrupts 0 >> Machine check exceptions 0 >> >> >> irqtop for cpu 5 >> >> eth0 257 >> Local timer interrupts 251 >> Rescheduling interrupts 237 >> Spurious interrupts 0 >> Machine check exceptions 0 >> >> So the timer interrupt for cpu4 is not completely dead but somehow >> broken. > > That is incredibly weird. > >> What can cause this problem? Any way to speed it up again? >> > > The host has 8 cpus and is only running this 6 vcpu guest, yes? The host is an dual quad core E5520 with hyperthrading enabled, so we see 2x4x2=16 CPUs on the host. The guest is started with 6 CPUs. > Can you confirm the other vcpus are ticking at 250 Hz? The irqtop shows different numbers for local timer interrupts on the other CPUs. The total number (summed up over all CPUs) varies between something like 700 and 1400. Any CPU can be down to 10 and next update up to 260. Only CPU4 stays at the 6 local timer interrupts. > > What does 'top' show running on cpu 4? Pressing 'f' 'j' will add a > last-used-cpu field in the display. The processes are not bound to a particular CPU, so the picture varies. Here are two shots: take1: 15 root RT -5 0 0 0 S 0 0.0 0:01.70 4 migration/4 16 root 15 -5 0 0 0 S 0 0.0 0:00.08 4 ksoftirqd/4 17 root RT -5 0 0 0 S 0 0.0 0:00.00 4 watchdog/4 25 root 15 -5 0 0 0 S 0 0.0 0:00.01 4 events/4 35 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kintegrityd/4 41 root 15 -5 0 0 0 S 0 0.0 0:00.03 4 kblockd/4 50 root 15 -5 0 0 0 S 0 0.0 0:00.90 4 ata/4 55 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kseriod 66 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 aio/4 73 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 crypto/4 80 root 15 -5 0 0 0 S 0 0.0 2:11.71 4 scsi_eh_1 87 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kmpathd/4 95 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kondemand/4 101 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kconservative/4 103 root 10 -10 0 0 0 S 0 0.0 0:00.00 4 krfcommd 681 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 686 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 691 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 737 root 15 -5 0 0 0 S 0 0.0 0:00.71 4 kjournald 826 root 16 -4 2100 452 312 S 0 0.0 0:00.14 4 udevd 1350 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kpsmoused 1444 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kgameportd 1718 root 15 -5 0 0 0 S 0 0.0 0:14.62 4 kjournald 2108 statd 20 0 2252 1152 760 S 0 0.0 0:02.66 4 rpc.statd 2117 root 15 -5 0 0 0 S 0 0.0 0:00.36 4 rpciod/4 2123 root 15 -5 0 0 0 S 0 0.0 0:06.61 4 nfsiod 2259 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty 2265 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty 2584 root 20 0 1860 656 604 S 0 0.0 0:01.92 4 automount 2712 mysql 20 0 146m 4580 3464 S 0 0.1 0:59.71 4 mysqld 2999 root 20 0 28944 12m 1704 S 0 0.2 1:23.04 4 console-kit-dae 3834 root 20 0 13476 2640 1572 S 0 0.0 0:00.03 4 smbd 4015 root 15 -5 0 0 0 S 0 0.0 7:22.33 4 nfsd 4017 root 15 -5 0 0 0 S 0 0.0 10:46.87 4 nfsd 4030 root 20 0 2644 1180 876 S 0 0.0 0:04.32 4 rpc.mountd 4134 root 20 0 6164 1672 1344 S 0 0.0 0:00.37 4 master 4311 root 20 0 5068 1796 1580 S 0 0.0 1:25.80 4 hald-addon-stor 4560 root 20 0 2092 728 572 S 0 0.0 0:01.78 4 cron 5053 www-data 20 0 33312 4092 1004 S 0 0.0 0:00.03 4 apache2 take 2: 15 root RT -5 0 0 0 S 0 0.0 0:01.70 4 migration/4 16 root 15 -5 0 0 0 S 0 0.0 0:00.08 4 ksoftirqd/4 17 root RT -5 0 0 0 S 0 0.0 0:00.00 4 watchdog/4 25 root 15 -5 0 0 0 S 0 0.0 0:00.01 4 events/4 35 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kintegrityd/4 41 root 15 -5 0 0 0 S 0 0.0 0:00.03 4 kblockd/4 50 root 15 -5 0 0 0 S 0 0.0 0:00.90 4 ata/4 55 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kseriod 66 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 aio/4 73 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 crypto/4 87 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kmpathd/4 95 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kondemand/4 101 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kconservative/4 103 root 10 -10 0 0 0 S 0 0.0 0:00.00 4 krfcommd 681 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 686 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 691 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kdmflush 737 root 15 -5 0 0 0 S 0 0.0 0:00.71 4 kjournald 826 root 16 -4 2100 452 312 S 0 0.0 0:00.14 4 udevd 1350 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kpsmoused 1444 root 15 -5 0 0 0 S 0 0.0 0:00.00 4 kgameportd 2091 daemon 20 0 1820 552 464 S 0 0.0 0:44.31 4 portmap 2108 statd 20 0 2252 1152 760 S 0 0.0 0:02.66 4 rpc.statd 2117 root 15 -5 0 0 0 S 0 0.0 0:00.36 4 rpciod/4 2123 root 15 -5 0 0 0 S 0 0.0 0:06.61 4 nfsiod 2259 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty 2265 root 20 0 1696 444 440 S 0 0.0 0:00.00 4 getty 2712 mysql 20 0 146m 4580 3464 S 0 0.1 0:59.72 4 mysqld 3834 root 20 0 13476 2640 1572 S 0 0.0 0:00.03 4 smbd 4017 root 15 -5 0 0 0 S 0 0.0 10:46.87 4 nfsd 4134 root 20 0 6164 1672 1344 S 0 0.0 0:00.37 4 master 4311 root 20 0 5068 1796 1580 S 0 0.0 1:25.80 4 hald-addon-stor 4560 root 20 0 2092 728 572 S 0 0.0 0:01.78 4 cron 5053 www-data 20 0 33312 4092 1004 S 0 0.0 0:00.03 4 apache2 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
