Possible nested virtualization issues? Guest panic + hung task on host

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm running qemu on a 4.8.x host which is also running under KVM
(openstack, probably ubuntu) on a server that i don't control. I've
had issues with this also on 4.7 and on a different physical host.

In guest i see this:

Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.540365] BUG: unable
to handle kernel paging request at 0000000008473000
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] IP:
[<ffffffff810a44bf>] search_module_extables+0x2f/0x70
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] PGD 0
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] Oops: 0000 [#1]
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] Modules
linked in: 9p fscache af_packet nft_limit nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_log_ipv6 nf_log_ipv4
nf_log_common nft_counter nft_l
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] CPU: 0 PID:
398 Comm: systemd-network Tainted: G        W       4.8.9-1-kvmnogr #1
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] Hardware
name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.10.1-20161122_114906-anatol 04/01/2014
Nov 30 14:09:50 foobar qemu-kvm-guest[59]: [21618.550075] task:
ffff8bce7f426e00 task.stack: ffff8bce7b0d4000
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] RIP:
0010:[<ffffffff810a44bf>]  [<ffffffff810a44bf>]
search_module_extables+0x2f/0x70
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] RSP:
0000:ffff8bce7b0d78a8  EFLAGS: 00010202
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] RAX:
ffffffffc0265348 RBX: ffffffffc0265340 RCX: ffffffff810ddb97
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] RDX:
ffffffff810a44bf RSI: ffffffff814a4110 RDI: ffffffff810a44bf
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] RBP:
ffff8bce7b0d78b8 R08: aaaaaaaaaaaaaaab R09: 0000000000000000
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] R10:
0000000000000046 R11: 0000000000000001 R12: ffffffff810a44bf
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] R13:
0000000000000000 R14: 000000000000000b R15: ffff8bce7f426e00
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] FS:
00007f024d2367c0(0000) GS:ffffffff8181f000(0000)
knlGS:0000000000000000
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] CS:  0010
DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] CR2:
0000000008473000 CR3: 0000000039210000 CR4: 00000000001406b0
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] Stack:
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075]
ffffffff810a44bf 000000000000000e ffff8bce7b0d78d0 ffffffff81070041
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075]
ffff8bce7b0d7a78 ffff8bce7b0d78f0 ffffffff8102a4e9 0000000000000000
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075]
ffff8bce7b0d7a78 ffff8bce7b0d7958 ffffffff810291b3 ffff8bce7fa84000
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075] Call Trace:
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810a44bf>] ? search_module_extables+0x2f/0x70
Nov 30 14:09:51 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81070041>] search_exception_tables+0x31/0x40
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8102a4e9>] fixup_exception+0x19/0x40
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810291b3>] no_context+0x43/0x310
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8102955a>] __bad_area_nosemaphore+0xda/0x1c0
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810296a0>] bad_area+0x40/0x50
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81029bd5>] __do_page_fault+0x335/0x3e0
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81029cd2>] do_page_fault+0x22/0x30
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81025a43>] do_async_page_fault+0x13/0x60
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff814a2998>] async_page_fault+0x28/0x30
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81253dda>] ? copy_user_generic_unrolled+0x1a/0xc0
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810a44bf>] ? search_module_extables+0x2f/0x70
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81070041>] search_exception_tables+0x31/0x40
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8102a4e9>] fixup_exception+0x19/0x40
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810291b3>] no_context+0x43/0x310
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811251b0>] ? alloc_debug_processing+0x70/0x1b0
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8102955a>] __bad_area_nosemaphore+0xda/0x1c0
Nov 30 14:09:52 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8113568f>] ? __check_object_size+0x4f/0x383
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810296a0>] bad_area+0x40/0x50
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81029bd5>] __do_page_fault+0x335/0x3e0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81029cd2>] do_page_fault+0x22/0x30
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81025a43>] do_async_page_fault+0x13/0x60
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff814a2998>] async_page_fault+0x28/0x30
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811830f0>] ? cn_esc_printf+0x110/0x110
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810645cf>] ? get_signal+0x17f/0x4a0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8100dcd2>] do_signal+0x32/0x5a0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff811045cb>] ? print_vma_addr+0x7b/0x100
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff8102953e>] ? __bad_area_nosemaphore+0xbe/0x1c0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81084e9a>] ? vprintk_default+0x1a/0x20
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810296a0>] ? bad_area+0x40/0x50
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81029bd5>] ? __do_page_fault+0x335/0x3e0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff81001052>] exit_to_usermode_loop+0x82/0xa0
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff810011b8>] prepare_exit_to_usermode+0x18/0x20
Nov 30 14:09:53 foobar qemu-kvm-guest[59]: [21618.550075]
[<ffffffff814a20e2>] retint_user+0x8/0x10
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21618.550075] Code: d3 78
00 48 3d 10 18 83 81 74 58 55 48 89 e5 41 54 49 89 fc 53 48 8d 58 f8
eb 10 48 8b 43 08 48 3d 10 18 83 81 48 8d 58 f8 74 31 <83> 3b 03 74 eb
8b 83 3c 0
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21618.550075] RIP
[<ffffffff810a44bf>] search_module_extables+0x2f/0x70
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21618.550075]  RSP
<ffff8bce7b0d78a8>
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21618.550075] CR2: 0000000008473000
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21623.232115] ---[ end
trace 73878b0600d58483 ]---
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21623.266619] Kernel panic
- not syncing: Fatal exception
Nov 30 14:09:54 foobar qemu-kvm-guest[59]: [21623.276559] Kernel
Offset: 0x0 from 0xffffffff81000000 (relocation range:
0xffffffff80000000-0xffffffffbfffffff)

and on the host

Nov 30 14:11:55 host.example.org kernel: INFO: task dnsmasq:538
blocked for more than 120 seconds.
Nov 30 14:11:55 host.example.org kernel:       Not tainted 4.8.9-1-kvmnogr #1
Nov 30 14:11:55 host.example.org kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 30 14:11:55 host.example.org kernel: dnsmasq         D
ffff9945b437fe98     0   538    413 0x00000180
Nov 30 14:11:55 host.example.org kernel:  ffff9945b437fe98
ffff9945b5814200 0000000000000000 ffff9945b4380000
Nov 30 14:11:55 host.example.org kernel:  ffff9945b437fef0
00000000ffffffff 00007ffe0839d2c0 00007fd9c36a1c00
Nov 30 14:11:55 host.example.org kernel:  ffff9945b437feb0
ffffffff8a49ec7e 00000000000000ed ffff9945b437ff38
Nov 30 14:11:55 host.example.org kernel: Call Trace:
Nov 30 14:11:55 host.example.org kernel:  [<ffffffff8a49ec7e>]
schedule+0x2e/0x80
Nov 30 14:11:55 host.example.org kernel:  [<ffffffff8a0258ac>]
kvm_async_pf_task_wait+0x17c/0x1c0
Nov 30 14:11:55 host.example.org kernel:  [<ffffffff8a025a87>]
do_async_page_fault+0x57/0x60
Nov 30 14:11:55 host.example.org kernel:  [<ffffffff8a4a2998>]
async_page_fault+0x28/0x30

I suppose my question is if this could have something to do with the
kernel itself rather than with the hoster doing weird things and if
this is worth debugging or not..and what can I do to provide more
information. Note that I can't reproduce this at will..it takes some
time and appears somewhat random
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [KVM ARM]     [KVM ia64]     [KVM ppc]     [Virtualization Tools]     [Spice Development]     [Libvirt]     [Libvirt Users]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite Questions]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux