On Wed, Jun 16, 2010 at 12:05:10PM +0200, Sebastian Hetze wrote: > Hi *, > > we recently have encountered kvm_queue_exception events in dmesg. > What is causing these exceptions? It appears that the guest is > at least suffering performance loss. Is the qemu-system-x86 hang > event linked to the exception? Is there anything we can do > to prevent these exceptions to happen? > What is your guest? It look like it tries to switch task and tss it tries to switch to is broken, so task switch emulation code encounters multiple exception and it complains about it. Theoretically we should abort task switch after fist exception condition (set_cr3: #GP, reserved bits), but error handling in task switch emulation code is far from perfect especially in such old kernel like 2.6.31. > Best regards, > > Sebastian > > --------------------------------- > [1811816.496620] set_cr3: #GP, reserved bits > [1811816.496624] ------------[ cut here ]------------ > [1811816.496640] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm > ]() > [1811816.496643] Hardware name: S5520HC > [1811816.496644] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv > 4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm > bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses encl > osure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor > [1811816.496669] Pid: 26170, comm: qemu-system-x86 Not tainted 2.6.31-14-server #48 > [1811816.496672] Call Trace: > [1811816.496682] [<ffffffffa0140651>] ? kvm_queue_exception_e+0x61/0x70 [kvm] > [1811816.496689] [<ffffffff8105f458>] warn_slowpath_common+0x78/0xd0 > [1811816.496692] [<ffffffff8105f4bf>] warn_slowpath_null+0xf/0x20 > [1811816.496701] [<ffffffffa0140651>] kvm_queue_exception_e+0x61/0x70 [kvm] > [1811816.496711] [<ffffffffa01406ff>] load_guest_segment_descriptor+0x9f/0xb0 [kvm] > [1811816.496721] [<ffffffffa0140753>] kvm_load_segment_descriptor+0x43/0x110 [kvm] > [1811816.496730] [<ffffffffa0140a20>] ? kvm_inject_gp+0x10/0x20 [kvm] > [1811816.496740] [<ffffffffa0140bed>] ? kvm_set_cr3+0x15d/0x170 [kvm] > [1811816.496749] [<ffffffffa0140d67>] load_state_from_tss32+0x167/0x240 [kvm] > [1811816.496758] [<ffffffffa0142a69>] kvm_task_switch_32+0x119/0x130 [kvm] > [1811816.496768] [<ffffffffa0142f93>] kvm_task_switch+0x153/0x300 [kvm] > [1811816.496777] [<ffffffffa0137df1>] ? gfn_to_hva+0x11/0x90 [kvm] > [1811816.496786] [<ffffffffa0138122>] ? kvm_read_guest_page+0x62/0x70 [kvm] > [1811816.496792] [<ffffffffa016cbab>] handle_task_switch+0x6b/0x160 [kvm_intel] > [1811816.496796] [<ffffffffa0169b45>] vmx_handle_exit+0xf5/0x280 [kvm_intel] > [1811816.496801] [<ffffffffa016d822>] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] > [1811816.496811] [<ffffffffa013ddbf>] vcpu_enter_guest+0x2cf/0x600 [kvm] > [1811816.496814] [<ffffffff8107a12e>] ? finish_wait+0x5e/0x80 > [1811816.496819] [<ffffffff81541710>] ? _spin_lock_irq+0x10/0x20 > [1811816.496822] [<ffffffff81541603>] ? __down_read+0xc3/0xce > [1811816.496825] [<ffffffff81079fa0>] ? autoremove_wake_function+0x0/0x40 > [1811816.496834] [<ffffffffa013e153>] __vcpu_run+0x63/0x330 [kvm] > [1811816.496844] [<ffffffffa01440ab>] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] > [1811816.496853] [<ffffffffa01367f2>] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] > [1811816.496856] [<ffffffff8106e99f>] ? dequeue_signal+0x9f/0x180 > [1811816.496859] [<ffffffff81131d41>] vfs_ioctl+0x31/0xa0 > [1811816.496862] [<ffffffff811321b3>] do_vfs_ioctl+0x373/0x400 > [1811816.496864] [<ffffffff811322d9>] sys_ioctl+0x99/0xa0 > [1811816.496862] [<ffffffff811321b3>] do_vfs_ioctl+0x373/0x400 > [1811816.496864] [<ffffffff811322d9>] sys_ioctl+0x99/0xa0 > [1811816.496867] [<ffffffff8106d3e9>] ? do_sigpending+0xa9/0xc0 > [1811816.496870] [<ffffffff81012082>] system_call_fastpath+0x16/0x1b > [1811816.496872] ---[ end trace c6ea0da2c3c9d0e4 ]--- > [1811816.496887] set_cr3: #GP, pdptrs reserved bits > [1811816.496889] ------------[ cut here ]------------ > [1811816.496898] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm]() > [1811816.496900] Hardware name: S5520HC > [1811816.496901] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses enclosure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor > [1811816.496923] Pid: 26170, comm: qemu-system-x86 Tainted: G W 2.6.31-14-server #48 > [1811816.496925] Call Trace: > [1811816.496934] [<ffffffffa0140651>] ? kvm_queue_exception_e+0x61/0x70 [kvm] > [1811816.496937] [<ffffffff8105f458>] warn_slowpath_common+0x78/0xd0 > [1811816.496940] [<ffffffff8105f4bf>] warn_slowpath_null+0xf/0x20 > [1811816.496950] [<ffffffffa0140651>] kvm_queue_exception_e+0x61/0x70 [kvm] > [1811816.496959] [<ffffffffa01406ff>] load_guest_segment_descriptor+0x9f/0xb0 [kvm] > [1811816.496969] [<ffffffffa0140753>] kvm_load_segment_descriptor+0x43/0x110 [kvm] > [1811816.496978] [<ffffffffa0140b52>] ? kvm_set_cr3+0xc2/0x170 [kvm] > [1811816.496988] [<ffffffffa0140d9c>] load_state_from_tss32+0x19c/0x240 [kvm] > [1811816.496997] [<ffffffffa0142a69>] kvm_task_switch_32+0x119/0x130 [kvm] > [1811816.497006] [<ffffffffa0142f93>] kvm_task_switch+0x153/0x300 [kvm] > [1811816.497015] [<ffffffffa0137df1>] ? gfn_to_hva+0x11/0x90 [kvm] > [1811816.497024] [<ffffffffa0138122>] ? kvm_read_guest_page+0x62/0x70 [kvm] > [1811816.497029] [<ffffffffa016cbab>] handle_task_switch+0x6b/0x160 [kvm_intel] > [1811816.497034] [<ffffffffa0169b45>] vmx_handle_exit+0xf5/0x280 [kvm_intel] > [1811816.497038] [<ffffffffa016d822>] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] > [1811816.497048] [<ffffffffa013ddbf>] vcpu_enter_guest+0x2cf/0x600 [kvm] > [1811816.497052] [<ffffffffa01680e3>] ? __vmx_load_host_state+0xc3/0x1e0 [kvm_intel] > [1811816.497055] [<ffffffff8107a12e>] ? finish_wait+0x5e/0x80 > [1811816.497059] [<ffffffff81036609>] ? default_spin_lock_flags+0x9/0x10 > [1811816.497062] [<ffffffff81541710>] ? _spin_lock_irq+0x10/0x20 > [1811816.497065] [<ffffffff81541603>] ? __down_read+0xc3/0xce > [1811816.497074] [<ffffffffa013e153>] __vcpu_run+0x63/0x330 [kvm] > [1811816.497083] [<ffffffffa01440ab>] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] > [1811816.497092] [<ffffffffa01367f2>] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] > [1811816.497095] [<ffffffff8106e99f>] ? dequeue_signal+0x9f/0x180 > [1811816.497098] [<ffffffff81131d41>] vfs_ioctl+0x31/0xa0 > [1811816.497100] [<ffffffff811321b3>] do_vfs_ioctl+0x373/0x400 > [1811816.497103] [<ffffffff811322d9>] sys_ioctl+0x99/0xa0 > [1811816.497105] [<ffffffff8106d3e9>] ? do_sigpending+0xa9/0xc0 > [1811816.497109] [<ffffffff81012082>] system_call_fastpath+0x16/0x1b > [1811816.497110] ---[ end trace c6ea0da2c3c9d0e5 ]--- > > ...... > > [3307007.208029] INFO: task qemu-system-x86:5327 blocked for more than 120 seconds. > [3307007.208176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [3307007.208350] qemu-system-x D 00000000ffffffff 0 5327 1 0x00000000 > [3307007.208356] ffff880bc80f3af8 0000000000000086 0000000000015880 0000000000015880 > [3307007.208364] 0000000000015880 0000000000015880 0000000000015880 0000000000015880 > [3307007.208369] 0000000000015880 ffff880c4f8fad60 ffff880654d48000 ffff880c4f8fb110 > [3307007.208374] Call Trace: > [3307007.208390] [<ffffffff8153f977>] io_schedule+0x37/0x50 > [3307007.208403] [<ffffffff81151d11>] dio_await_completion+0x71/0x100 > [3307007.208415] [<ffffffff81152136>] direct_io_worker+0x2b6/0x390 > [3307007.208425] [<ffffffff81152497>] __blockdev_direct_IO+0x287/0x3f0 > [3307007.208436] [<ffffffff8114f0c0>] ? blkdev_get_blocks+0x0/0xc0 > [3307007.208445] [<ffffffff81150139>] blkdev_direct_IO+0x49/0x50 > [3307007.208454] [<ffffffff8114f0c0>] ? blkdev_get_blocks+0x0/0xc0 > [3307007.208468] [<ffffffff810de7d8>] generic_file_direct_write+0xc8/0x230 > [3307007.208479] [<ffffffff810df19e>] __generic_file_aio_write_nolock+0x36e/0x470 > [3307007.208494] [<ffffffff8107d7af>] ? hrtimer_try_to_cancel+0x3f/0x90 > [3307007.208504] [<ffffffff810df2d7>] generic_file_aio_write_nolock+0x37/0xa0 > [3307007.208516] [<ffffffff811224e1>] do_sync_write+0xf1/0x140 > [3307007.208527] [<ffffffff81079fa0>] ? autoremove_wake_function+0x0/0x40 > [3307007.208539] [<ffffffff81089e93>] ? futex_wake+0x113/0x130 > [3307007.208552] [<ffffffff81226871>] ? security_file_permission+0x11/0x20 > [3307007.208562] [<ffffffff81122b5b>] vfs_write+0xcb/0x130 > [3307007.208571] [<ffffffff81122c52>] sys_pwrite64+0x92/0xa0 > [3307007.208582] [<ffffffff81012082>] system_call_fastpath+0x16/0x1b > [3307007.208592] INFO: task qemu-system-x86:5320 blocked for more than 120 seconds. > [3307007.208724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [3307007.208858] qemu-system-x D 0000000000000000 0 5320 1 0x00400000 > [3307007.208863] ffff880bb3d45af8 0000000000000082 0000000000015880 0000000000015880 > [3307007.208868] 0000000000015880 0000000000015880 0000000000015880 0000000000015880 > [3307007.208873] 0000000000015880 ffff880c4fd2c410 ffff880654c94410 ffff880c4fd2c7c0 > [3307007.208877] Call Trace: > [3307007.208882] [<ffffffff8153f977>] io_schedule+0x37/0x50 > [3307007.208886] [<ffffffff81151d11>] dio_await_completion+0x71/0x100 > [3307007.208890] [<ffffffff81152136>] direct_io_worker+0x2b6/0x390 > [3307007.208895] [<ffffffff81152497>] __blockdev_direct_IO+0x287/0x3f0 > [3307007.208904] [<ffffffff8114f0c0>] ? blkdev_get_blocks+0x0/0xc0 > [3307007.208916] [<ffffffff81150139>] blkdev_direct_IO+0x49/0x50 > [3307007.208886] [<ffffffff81151d11>] dio_await_completion+0x71/0x100 > [3307007.208890] [<ffffffff81152136>] direct_io_worker+0x2b6/0x390 > [3307007.208895] [<ffffffff81152497>] __blockdev_direct_IO+0x287/0x3f0 > [3307007.208904] [<ffffffff8114f0c0>] ? blkdev_get_blocks+0x0/0xc0 > [3307007.208916] [<ffffffff81150139>] blkdev_direct_IO+0x49/0x50 > [3307007.208926] [<ffffffff8114f0c0>] ? blkdev_get_blocks+0x0/0xc0 > [3307007.208936] [<ffffffff810de7d8>] generic_file_direct_write+0xc8/0x230 > [3307007.208949] [<ffffffff810df19e>] __generic_file_aio_write_nolock+0x36e/0x470 > [3307007.208960] [<ffffffff8107d7af>] ? hrtimer_try_to_cancel+0x3f/0x90 > [3307007.208971] [<ffffffff810df2d7>] generic_file_aio_write_nolock+0x37/0xa0 > [3307007.208981] [<ffffffff811224e1>] do_sync_write+0xf1/0x140 > [3307007.208991] [<ffffffff81079fa0>] ? autoremove_wake_function+0x0/0x40 > [3307007.209002] [<ffffffff81089e93>] ? futex_wake+0x113/0x130 > [3307007.209012] [<ffffffff81226871>] ? security_file_permission+0x11/0x20 > [3307007.209023] [<ffffffff81122b5b>] vfs_write+0xcb/0x130 > [3307007.209033] [<ffffffff81122c52>] sys_pwrite64+0x92/0xa0 > [3307007.209043] [<ffffffff81012082>] system_call_fastpath+0x16/0x1b > > ...... > > [4837287.970390] set_cr3: #GP, reserved bits > [4837287.970394] ------------[ cut here ]------------ > [4837287.970411] WARNING: at /usr/src/linux-2.6.31/arch/x86/kvm/x86.c:202 kvm_queue_exception_e+0x61/0x70 [kvm]() > [4837287.970413] Hardware name: S5520HC > [4837287.970414] Modules linked in: drbd tun ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp ppdev iptable_filter ip_tables lp parport x_tables kvm_intel kvm bridge stp bonding e100 via_rhine 3c59x 8139too mii shpchp iTCO_wdt iTCO_vendor_support pcspkr joydev ses enclosure usbhid bnx2 aacraid e1000e igb dca fbcon tileblit font bitblit softcursor > [4837287.970440] Pid: 6524, comm: qemu-system-x86 Tainted: G W 2.6.31-14-server #48 > [4837287.970442] Call Trace: > [4837287.970453] [<ffffffffa0140651>] ? kvm_queue_exception_e+0x61/0x70 [kvm] > [4837287.970459] [<ffffffff8105f458>] warn_slowpath_common+0x78/0xd0 > [4837287.970462] [<ffffffff8105f4bf>] warn_slowpath_null+0xf/0x20 > [4837287.970472] [<ffffffffa0140651>] kvm_queue_exception_e+0x61/0x70 [kvm] > [4837287.970481] [<ffffffffa01406ff>] load_guest_segment_descriptor+0x9f/0xb0 [kvm] > [4837287.970491] [<ffffffffa0140753>] kvm_load_segment_descriptor+0x43/0x110 [kvm] > [4837287.970501] [<ffffffffa0140bed>] ? kvm_set_cr3+0x15d/0x170 [kvm] > [4837287.970510] [<ffffffffa0140d9c>] load_state_from_tss32+0x19c/0x240 [kvm] > [4837287.970520] [<ffffffffa0142a69>] kvm_task_switch_32+0x119/0x130 [kvm] > [4837287.970529] [<ffffffffa0142f93>] kvm_task_switch+0x153/0x300 [kvm] > [4837287.970535] [<ffffffffa016cbab>] handle_task_switch+0x6b/0x160 [kvm_intel] > [4837287.970540] [<ffffffffa0169b45>] vmx_handle_exit+0xf5/0x280 [kvm_intel] > [4837287.970545] [<ffffffffa016d822>] ? vmx_vcpu_run+0x292/0x856 [kvm_intel] > [4837287.970554] [<ffffffffa013ddbf>] vcpu_enter_guest+0x2cf/0x600 [kvm] > [4837287.970558] [<ffffffff8107a12e>] ? finish_wait+0x5e/0x80 > [4837287.970563] [<ffffffff81541710>] ? _spin_lock_irq+0x10/0x20 > [4837287.970566] [<ffffffff81541603>] ? __down_read+0xc3/0xce > [4837287.970568] [<ffffffff81079fa0>] ? autoremove_wake_function+0x0/0x40 > [4837287.970578] [<ffffffffa013e153>] __vcpu_run+0x63/0x330 [kvm] > [4837287.970587] [<ffffffffa01440ab>] kvm_arch_vcpu_ioctl_run+0x8b/0x1f0 [kvm] > [4837287.970596] [<ffffffffa01367f2>] kvm_vcpu_ioctl+0x2e2/0x630 [kvm] > [4837287.970599] [<ffffffff8106e99f>] ? dequeue_signal+0x9f/0x180 > [4837287.970603] [<ffffffff81012a8e>] ? common_interrupt+0xe/0x13 > [4837287.970607] [<ffffffff81131d41>] vfs_ioctl+0x31/0xa0 > [4837287.970609] [<ffffffff811321b3>] do_vfs_ioctl+0x373/0x400 > [4837287.970612] [<ffffffff811322db>] ? sys_ioctl+0x9b/0xa0 > [4837287.970614] [<ffffffff811322d9>] sys_ioctl+0x99/0xa0 > [4837287.970617] [<ffffffff8106d3e9>] ? do_sigpending+0xa9/0xc0 > [4837287.970620] [<ffffffff81012082>] system_call_fastpath+0x16/0x1b > [4837287.970622] ---[ end trace c6ea0da2c3c9d0e6 ]--- > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
