On 01/31/2012 04:06 PM, Bram De Wilde wrote:
> Hi,
>
> We are setting up a ubuntu / openstack / KVM based private cloud. Of the 3 servers involved in this configuration 1 has been crashing frequently over the last month.
> Since it is the only server responsible for the virtualization (so the only one running kvm virtual machines) my guess was to find some help with you guys. We are running the same install on 2 other machines (not running virtual machines!) who have been completely stable over the last three months, so a hardware issue might also be at play here?
>
> I have collected some stack traces over the last month and was hoping one of you might point us in the right direction as to what is going wrong on our system.
> We are running a default ubuntu 11.10 server install. The kernel has been upgraded from the 3.0.0-12 release to the 3.0.0-14 release over the last month, but we keep getting these crashes. We are now running the 3.2.2-030202-generic hoping that this is a kernel problem, and it is fixed in this latest version.
>
> If any additional information is required or I should file an official bug with this information, please let me know...
File a bug please.
Meanwhile:
> Jan 27 13:41:28 cmggcn01 kernel: [871350.761867] general protection fault: 0000 [#2] SMP
> Jan 27 13:41:28 cmggcn01 kernel: [871350.790117] CPU 14
> Jan 27 13:41:28 cmggcn01 kernel: [871350.790387] Modules linked in: btrfs zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs reiserfs ebt_arp ebt_ip 8021q garp ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm nbd vesafb ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi psmouse dcdbas dm_multipath serio_raw joydev ghes hed acpi_power_meter bonding lp parport i7core_edac edac_core ses enclosure usbhid hid megaraid_sas bnx2
> Jan 27 13:41:28 cmggcn01 kernel: [871351.005151]
> Jan 27 13:41:28 cmggcn01 kernel: [871351.036187] Pid: 90, comm: kswapd0 Tainted: G D 3.0.0-14-server #23-Ubuntu Dell Inc. PowerEdge R710/0MD99X
> Jan 27 13:41:28 cmggcn01 kernel: [871351.072809] RIP: 0010:[<ffffffffa01a5890>] [<ffffffffa01a5890>] kvm_unmap_rmapp+0x20/0x60 [kvm]
> Jan 27 13:41:28 cmggcn01 kernel: [871351.105190] RSP: 0018:ffff8817f3e27a60 EFLAGS: 00010202
> Jan 27 13:41:28 cmggcn01 kernel: [871351.141329] RAX: 00008817f5d067f8 RBX: ffffc9001fd41ff8 RCX: ffffffffa01a58d0
> Jan 27 13:41:28 cmggcn01 kernel: [871351.179076] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00008817f5d067f8
> Jan 27 13:41:28 cmggcn01 kernel: [871351.212086] RBP: ffff8817f3e27a80 R08: ffff8817f315b3e0 R09: 0000000000000100
> Jan 27 13:41:28 cmggcn01 kernel: [871351.245788] R10: 000000000000000e R11: 0000000000000002 R12: ffff8817f2f0c000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.277514] R13: 0000000000000000 R14: ffff880be235e000 R15: 00000000000d3cff
> Jan 27 13:41:28 cmggcn01 kernel: [871351.308421] FS: 0000000000000000(0000) GS:ffff88183fce0000(0000) knlGS:0000000000000000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.339685] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Jan 27 13:41:28 cmggcn01 kernel: [871351.370089] CR2: 00007f8836442000 CR3: 0000000001c03000 CR4: 00000000000026e0
> Jan 27 13:41:28 cmggcn01 kernel: [871351.399771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.428208] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Jan 27 13:41:28 cmggcn01 kernel: [871351.456153] Process kswapd0 (pid: 90, threadinfo ffff8817f3e26000, task ffff8817f63ac560)
> Jan 27 13:41:28 cmggcn01 kernel: [871351.484943] Stack:
> Jan 27 13:41:28 cmggcn01 kernel: [871351.512984] 0000000000000000 ffffc9001fd41ff8 0000000000000001 00007f834a87e000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.542025] ffff8817f3e27aa0 ffffffffa01a5945 ffff880be235e060 0000000000000001
> Jan 27 13:41:28 cmggcn01 kernel: [871351.571050] ffff8817f3e27b10 ffffffffa01a1dd9 ffff8817f3e27ae0 ffffffffa01a58d0
<snip>
> Jan 27 13:41:28 cmggcn01 kernel: [871352.080582] Code: e7 d0 e8 e0 66 90 e9 a2 fe ff ff 55 48 89 e5 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 45 31 ed 49 89 fc 48 89 f3 eb 20 0f 1f 00 <f6> 00 01 74 35 48 8b 15 74 7a 02 00 48 89 c6 4c 89 e7 41 bd 01
0: e8 e0 66 90 e9 callq 0xffffffffe99066e5
5: a2 fe ff ff 55 48 89 mov %al,0x41e5894855fffffe
c: e5 41
e: 55 push %rbp
f: 41 54 push %r12
11: 53 push %rbx
12: 48 83 ec 08 sub $0x8,%rsp
16: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax
1b: 45 31 ed xor %r13d,%r13d
1e: 49 89 fc mov %rdi,%r12
21: 48 89 f3 mov %rsi,%rbx
24: eb 20 jmp 0x46
26: 0f 1f 00 nopl (%rax)
29: f6 00 01 testb $0x1,(%rax)
^ dies here, %rax is non-canonical.
2c: 74 35 je 0x63
2e: 48 8b 15 74 7a 02 00 mov 0x27a74(%rip),%rdx # 0x27aa9
35: 48 89 c6 mov %rax,%rsi
38: 4c 89 e7 mov %r12,%rdi
static int kvm_unmap_rmapp(struct kvm *kvm, unsigned long *rmapp,
unsigned long data)
{
u64 *spte;
int need_tlb_flush = 0;
while ((spte = rmap_next(kvm, rmapp, NULL))) {
BUG_ON(!(*spte & PT_PRESENT_MASK));
^ here, when fetching *spte.
rmap_printk("kvm_rmap_unmap_hva: spte %p %llx\n", spte, *spte);
drop_spte(kvm, spte);
need_tlb_flush = 1;
}
return need_tlb_flush;
Looks like a use-after-free with the two bytes at offset 6 zeroed.
If this is reproducible, please rerun with the host kernel parameter
slub_debug=FZPU.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
