host crashes "unable to handle paging request"

[Raphael Bauduin <rblists@xxxxxxxxx>]

Hi,

we have regular crashed of a kvm host with the error "unable to handle paging request".

Can this be due to memory over-commitment even if some memory is still used by the kernel for caches and buffers?  (collectd graph shows no free memory, with 15G used, very little buffers, and 1G cache). There are 32GB of swap, of which only 150MB are used.

I suspect might be the direction to search to find the cause, but would be happy to learn from people versed in the kernel behaviour to confirm or reject my hypothesis. Below is the full error.

Thanks!

Raph



745 Mar 23 14:27:37 sMaster01 kernel: [241450.355339] BUG: unable to handle kernel paging request at ffff8804c001fade
 746 Mar 23 14:27:37 sMaster01 kernel: [241450.355384] IP: [<ffffffff8117e9e9>] bio_check_eod+0x29/0xcd
 747 Mar 23 14:27:37 sMaster01 kernel: [241450.355433] PGD 1002063 PUD 0
 748 Mar 23 14:27:37 sMaster01 kernel: [241450.355464] Oops: 0000 [#1] SMP
 749 Mar 23 14:27:37 sMaster01 kernel: [241450.355496] last sysfs file: /sys/devices/system/cpu/cpu15/

topology/thread_siblings
 750 Mar 23 14:27:37 sMaster01 kernel: [241450.355551] CPU 4
 751 Mar 23 14:27:37 sMaster01 kernel: [241450.355577] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp kvm_amd kvm ip6table_filter ip6_tables iptable_fi     lter ip_tables x_tables tun nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc bridge stp bonding dm_round_robin dm_multipath scsi_dh loop snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw evdev tpm_tis tpm tpm_bios p     smouse pcspkr amd64_edac_mod edac_core button edac_mce_amd shpchp i2c_piix4 container pci_hotplug i2c_core processor ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sd_mod crc_t10dif mptsas mptscsih mptbase lpfc      ehci_hcd scsi_transport_fc tg3 scsi_tgt scsi_transport_sas ohci_hcd libphy scsi_mod usbcore nls_base thermal fan thermal_sys [last unloaded: scsi_wait_scan]
 752 Mar 23 14:27:37 sMaster01 kernel: [241450.356084] Pid: 3557, comm: kjournald Not tainted 2.6.32.61vanilla #1 PRIMERGY BX630 S2
 753 Mar 23 14:27:37 sMaster01 kernel: [241450.356141] RIP: 0010:[<ffffffff8117e9e9>]  [<ffffffff8117e9e9>] bio_check_eod+0x29/0xcd
 754 Mar 23 14:27:37 sMaster01 kernel: [241450.356196] RSP: 0018:ffff8804229abba0  EFLAGS: 00010202
 755 Mar 23 14:27:37 sMaster01 kernel: [241450.356228] RAX: ffff8804c001fad6 RBX: ffff8802e7235080 RCX: 00011200061e5110
 756 Mar 23 14:27:37 sMaster01 kernel: [241450.356279] RDX: 0000000000000008 RSI: 0000000000000008 RDI: ffff8802e7235080
 757 Mar 23 14:27:37 sMaster01 kernel: [241450.356331] RBP: ffff8802e7235080 R08: 0000000000000000 R09: ffff880425c54c00
 758 Mar 23 14:27:37 sMaster01 kernel: [241450.356383] R10: 0000000000000003 R11: 00000000022e539e R12: ffff8802e7235080
 759 Mar 23 14:27:37 sMaster01 kernel: [241450.356434] R13: ffff8802e7235080 R14: ffff880425c54c00 R15: ffff8802e6281850
 760 Mar 23 14:27:37 sMaster01 kernel: [241450.356486] FS:  00007faa6a757820(0000) GS:ffff88000fc80000(0000) knlGS:0000000000000000
 761 Mar 23 14:27:37 sMaster01 kernel: [241450.356540] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
 762 Mar 23 14:27:37 sMaster01 kernel: [241450.356573] CR2: ffff8804c001fade CR3: 00000000cc11f000 CR4: 00000000000006e0
 763 Mar 23 14:27:37 sMaster01 kernel: [241450.356628] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 764 Mar 23 14:27:37 sMaster01 kernel: [241450.356681] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 765 Mar 23 14:27:37 sMaster01 kernel: [241450.356733] Process kjournald (pid: 3557, threadinfo ffff8804229aa000, task ffff88041490a300)
 766 Mar 23 14:27:37 sMaster01 kernel: [241450.356788] Stack:
 767 Mar 23 14:27:37 sMaster01 kernel: [241450.356812]  ffff880415382c00 0000000100000285 ffff8804229abfd8 0000000000005186
 768 Mar 23 14:27:37 sMaster01 kernel: [241450.356852] <0> 0000000000000000 000000000f1c2776 ffff8804128efa38 ffff8802e7235080
 769 Mar 23 14:27:37 sMaster01 kernel: [241450.356913] <0> ffff8802e7235080 ffff8802e7235080 ffff8800cdacae40 ffffffff8117eb5a
 770 Mar 23 14:27:37 sMaster01 kernel: [241450.356993] Call Trace:
 771 Mar 23 14:27:37 sMaster01 kernel: [241450.357021]  [<ffffffff8117eb5a>] ? generic_make_request+0xcd/0x2f9
 772 Mar 23 14:27:37 sMaster01 kernel: [241450.357058]  [<ffffffff810b6034>] ? mempool_alloc+0x55/0x106
 773 Mar 23 14:27:37 sMaster01 kernel: [241450.357091]  [<ffffffff8117ee5c>] ? submit_bio+0xd6/0xf2
 774 Mar 23 14:27:37 sMaster01 kernel: [241450.357125]  [<ffffffff8110d83f>] ? submit_bh+0xf5/0x115
 775 Mar 23 14:27:37 sMaster01 kernel: [241450.357158]  [<ffffffff8110edc0>] ? sync_dirty_buffer+0x51/0x93
 776 Mar 23 14:27:37 sMaster01 kernel: [241450.357196]  [<ffffffffa01727c7>] ? journal_commit_transaction+0xaa6/0xe4f [jbd]
 777 Mar 23 14:27:37 sMaster01 kernel: [241450.357252]  [<ffffffffa0175194>] ? kjournald+0xdf/0x226 [jbd]
 778 Mar 23 14:27:37 sMaster01 kernel: [241450.357288]  [<ffffffff810651de>] ? autoremove_wake_function+0x0/0x2e
 779 Mar 23 14:27:37 sMaster01 kernel: [241450.357324]  [<ffffffffa01750b5>] ? kjournald+0x0/0x226 [jbd]
 780 Mar 23 14:27:37 sMaster01 kernel: [241450.357357]  [<ffffffff81064f11>] ? kthread+0x79/0x81
 781 Mar 23 14:27:37 sMaster01 kernel: [241450.357391]  [<ffffffff81011baa>] ? child_rip+0xa/0x20
 782 Mar 23 14:27:37 sMaster01 kernel: [241450.357425]  [<ffffffff81016568>] ? read_tsc+0xa/0x20
 783 Mar 23 14:27:37 sMaster01 kernel: [241450.357456]  [<ffffffff81064e98>] ? kthread+0x0/0x81
 784 Mar 23 14:27:37 sMaster01 kernel: [241450.357487]  [<ffffffff81011ba0>] ? child_rip+0x0/0x20
 785 Mar 23 14:27:37 sMaster01 kernel: [241450.357517] Code: 5c c3 41 55 49 89 fd 41 54 55 53 48 83 ec 38 65 48 8b 04 25 28 00 00 00 48 89 44 24 28 31 c0 85 f6 0f 84 86 00 00 00 48 8b 47 10 <48> 8b 40 08 48 8b 40 68 48 c1 f8 09 74 74 89      f2 48 8b 0f 48 39
 786 Mar 23 14:27:37 sMaster01 kernel: [241450.357738] RIP  [<ffffffff8117e9e9>] bio_check_eod+0x29/0xcd
 787 Mar 23 14:27:37 sMaster01 kernel: [241450.357772]  RSP <ffff8804229abba0>
 788 Mar 23 14:27:37 sMaster01 kernel: [241450.357799] CR2: ffff8804c001fade
 789 Mar 23 14:27:37 sMaster01 kernel: [241450.358183] ---[ end trace 608fcf1f5a482549 ]---

--
Web database: http://www.myowndb.com
Free Software Developers Meeting: http://www.fosdem.org

_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users