On Tue, May 5, 2009 at 3:29 PM, Justin Dossey <jbd@xxxxxxxxxxxxxxxx> wrote: > On Mon, May 4, 2009 at 3:40 PM, Marcelo Tosatti <mtosatti@xxxxxxxxxx> wrote: >> Justin, >> >> On Sun, May 03, 2009 at 11:40:47AM -0700, Justin Dossey wrote: > [snip] >> >> Seems to be an issue with paravirt mmu. Do you happen to have >> CONFIG_DEBUG_PAGEALLOC turned on your guests? > > I don't, as my VMs are in production use. To find the source of this > issue, I can turn it on though. > > While I'm at it, are there any other kernel features I should enable? > I went ahead and recompiled with CONFIG_DEBUG_PAGEALLOC enabled. Here's the panic (77 seconds after boot!) [ 76.911884] BUG: unable to handle kernel paging request at f4d17000 [ 76.915076] IP: [<c02856a9>] __slab_alloc+0x217/0x42f [ 76.917161] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 76.919309] last sysfs file: /sys/kernel/uevent_seqnum [ 76.920015] Modules linked in: [ 76.920015] [ 76.920015] Pid: 4632, comm: ruby18 Not tainted (2.6.28-gentoo-r4 #2) [ 76.920015] EIP: 0060:[<c02856a9>] EFLAGS: 00210086 CPU: 0 [ 76.920015] EIP is at __slab_alloc+0x217/0x42f [ 76.920015] EAX: c0761564 EBX: c1abb740 ECX: c0761564 EDX: 00000000 [ 76.920015] ESI: f4d17800 EDI: f4d17000 EBP: f72e9b8c ESP: f72e9b6c [ 76.920015] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 76.920015] Process ruby18 (pid: 4632, ti=f72e8000 task=f721d8e0 task.ti=f72e8000) [ 76.920015] Stack: [ 76.920015] ffffffff 00000020 c0761564 00000000 f4d10000 00000000 00200282 c0761564 [ 76.920015] f72e9bb4 c02867a5 c04b6782 c2a202a8 c04b6782 00000020 00000800 f70583c0 [ 76.920015] 00000600 f70583c0 f72e9bd4 c04b5948 00000000 00000020 c0760f58 00000020 [ 76.920015] Call Trace: [ 76.920015] [<c02867a5>] ? __kmalloc_track_caller+0x89/0xda [ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c04b5948>] ? __alloc_skb+0x4f/0xfb [ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c0452583>] ? try_fill_recv+0x30/0x177 [ 76.920015] [<c04b37a1>] ? sock_def_readable+0x5e/0x63 [ 76.920015] [<c0453237>] ? virtnet_poll+0x25c/0x309 [ 76.920015] [<c04bc32e>] ? net_rx_action+0xbd/0x1ea [ 76.920015] [<c022b2ed>] ? __do_softirq+0x83/0x12e [ 76.920015] [<c022b3e0>] ? do_softirq+0x48/0x57 [ 76.920015] [<c022b6fa>] ? irq_exit+0x38/0x6d [ 76.920015] [<c0205868>] ? do_IRQ+0x96/0xae [ 76.920015] [<c020471b>] ? common_interrupt+0x23/0x28 [ 76.920015] [<c0274379>] ? copy_page_range+0x25c/0x51e [ 76.920015] [<c0225731>] ? dup_mm+0x22a/0x30c [ 76.920015] [<c0226141>] ? copy_process+0x906/0x1026 [ 76.920015] [<c02269a0>] ? do_fork+0xd6/0x21f [ 76.920015] [<c03ae3e0>] ? copy_to_user+0x2a/0x36 [ 76.920015] [<c0202236>] ? sys_clone+0x25/0x2a [ 76.920015] [<c0203bd2>] ? syscall_call+0x7/0xb [ 76.920015] Code: c1 e9 02 f3 ab f6 c2 02 74 02 66 ab f6 c2 01 74 01 aa 8b 7d f0 89 fe eb 19 8b 45 e8 89 f9 89 da e8 4d ee ff ff 8b 4d e8 03 79 0c <89> 37 89 f7 03 71 04 8b 55 e8 0f b7 43 0a 0f af 42 04 03 45f0 [ 76.920015] EIP: [<c02856a9>] __slab_alloc+0x217/0x42f SS:ESP 0068:f72e9b6c [ 76.920015] Kernel panic - not syncing: Fatal exception in interrupt [ 76.920015] ------------[ cut here ]------------ [ 76.920015] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x33/0x185() [ 76.920015] Modules linked in: [ 76.920015] Pid: 4632, comm: ruby18 Tainted: G D 2.6.28-gentoo-r4 #2 [ 76.920015] Call Trace: [ 76.920015] [<c022735a>] warn_on_slowpath+0x41/0x5b [ 76.920015] [<c03ada5e>] ? __delay+0x9/0xb [ 76.920015] [<c0405af6>] ? serial8250_console_write+0xd6/0xea [ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [<c0586f9c>] ? _spin_unlock+0x1d/0x20 [ 76.920015] [<c03b1457>] ? _raw_spin_lock+0x60/0xf5 [ 76.920015] [<c023af8e>] ? up+0x2b/0x2f [ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [<c022797f>] ? release_console_sem+0x185/0x198 [ 76.920015] [<c03b12f7>] ? _raw_spin_unlock+0x8e/0x92 [ 76.920015] [<c022798a>] ? release_console_sem+0x190/0x198 [ 76.920015] [<c0246630>] smp_call_function_mask+0x33/0x185 [ 76.920015] [<c02100e4>] ? stop_this_cpu+0x0/0x50 [ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa [ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa [ 76.920015] [<c024f4cd>] ? crash_kexec+0x9f/0xa7 [ 76.920015] [<c0585ca3>] ? mutex_unlock+0x8/0xa [ 76.920015] [<c024f4cd>] ? crash_kexec+0x9f/0xa7 [ 76.920015] [<c0246794>] smp_call_function+0x12/0x14 [ 76.920015] [<c0210183>] native_smp_send_stop+0x1b/0x45 [ 76.920015] [<c02273d0>] panic+0x4e/0xef [ 76.920015] [<c0587e2e>] oops_end+0x8f/0xa3 [ 76.920015] [<c0205c87>] die+0x57/0x5f [ 76.920015] [<c0589477>] do_page_fault+0x63e/0x6fa [ 76.920015] [<c0268c84>] ? __alloc_pages_internal+0xad/0x33a [ 76.920015] [<c0588e39>] ? do_page_fault+0x0/0x6fa [ 76.920015] [<c05874fa>] error_code+0x72/0x78 [ 76.920015] [<c02856a9>] ? __slab_alloc+0x217/0x42f [ 76.920015] [<c02867a5>] __kmalloc_track_caller+0x89/0xda [ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c04b6782>] ? __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c04b5948>] __alloc_skb+0x4f/0xfb [ 76.920015] [<c04b6782>] __netdev_alloc_skb+0x17/0x34 [ 76.920015] [<c0452583>] try_fill_recv+0x30/0x177 [ 76.920015] [<c04b37a1>] ? sock_def_readable+0x5e/0x63 [ 76.920015] [<c0453237>] virtnet_poll+0x25c/0x309 [ 76.920015] [<c04bc32e>] net_rx_action+0xbd/0x1ea [ 76.920015] [<c022b2ed>] __do_softirq+0x83/0x12e [ 76.920015] [<c022b3e0>] do_softirq+0x48/0x57 [ 76.920015] [<c022b6fa>] irq_exit+0x38/0x6d [ 76.920015] [<c0205868>] do_IRQ+0x96/0xae [ 76.920015] [<c020471b>] common_interrupt+0x23/0x28 [ 76.920015] [<c0274379>] ? copy_page_range+0x25c/0x51e [ 76.920015] [<c0225731>] dup_mm+0x22a/0x30c [ 76.920015] [<c0226141>] copy_process+0x906/0x1026 [ 76.920015] [<c02269a0>] do_fork+0xd6/0x21f [ 76.920015] [<c03ae3e0>] ? copy_to_user+0x2a/0x36 [ 76.920015] [<c0202236>] sys_clone+0x25/0x2a [ 76.920015] [<c0203bd2>] syscall_call+0x7/0xb [ 76.920015] ---[ end trace 4656f6f379527461 ]--- -- Justin Dossey -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
