On Mon, Aug 8, 2016 at 9:57 PM, Victor Payno <vpayno@xxxxxxxxxx> wrote: > We have another problem where an RBD client was killed when an OSD was > killed by the OOM on a server. The servers have 4.4.16 kernels. > > ams2 login: [789881.620147] ------------[ cut here ]------------ > [789881.625094] kernel BUG at drivers/block/rbd.c:4638! > [789881.630311] invalid opcode: 0000 [#1] SMP > [789881.634650] Modules linked in: rbd libceph sg rpcsec_gss_krb5 > xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw > iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal > gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel > [789881.661718] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G > O 4.7.0-vanilla-ams-3 #1 > [789881.671091] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014 > [789881.678212] Workqueue: ceph-watch-notify do_watch_notify [libceph] > [789881.684814] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti: > ffff8803f0c90000 > [789881.692802] RIP: 0010:[<ffffffffa016d1c9>] [<ffffffffa016d1c9>] > rbd_dev_header_info+0x5a9/0x940 [rbd] > [789881.702702] RSP: 0018:ffff8803f0c93d30 EFLAGS: 00010286 > [789881.708344] RAX: 0000000000000077 RBX: ffff8802a6a63800 RCX: > 0000000000000000 > [789881.715985] RDX: 0000000000000077 RSI: ffff88041fd0dd08 RDI: > ffff88041fd0dd08 > [789881.723625] RBP: ffff8803f0c93d98 R08: 0000000000000030 R09: > 0000000000000000 > [789881.731261] R10: 0000000000000000 R11: 0000000000004479 R12: > ffff8800d6eaf000 > [789881.738899] R13: ffff8802a6a639b0 R14: 0000000000000000 R15: > ffff880327e6e780 > [789881.746533] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000) > knlGS:0000000000000000 > [789881.755120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [789881.761197] CR2: 00007fbb18242838 CR3: 0000000001e07000 CR4: > 00000000001406e0 > [789881.768846] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [789881.776482] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [789881.784118] Stack: > [789881.786457] ffffffff8113a91a ffff88032069ea00 ffff88041fd17ef0 > ffff88041fd17ef0 > [789881.794713] ffff88041fd17ef0 0000000000030289 ffff8803f0c93dd8 > ffffffff8113d968 > [789881.802965] ffff8802a6a63800 ffff8800d6eaf000 ffff8802a6a639b0 > 0000000000000000 > [789881.811207] Call Trace: > [789881.813988] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110 > [789881.819810] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150 > [789881.826321] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd] > [789881.832760] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd] > [789881.838930] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph] > [789881.845706] [<ffffffff811258e9>] process_one_work+0x149/0x3c0 > [789881.856639] [<ffffffff81125bae>] worker_thread+0x4e/0x490 > [789881.862453] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0 > [789881.868823] [<ffffffff8112b1e9>] kthread+0xc9/0xe0 > [789881.874033] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40 > [789881.879764] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170 > [789881.886618] Code: 0b 44 8b 6d b8 e9 1d ff ff ff 48 c7 c1 f0 00 17 > a0 ba 1e 12 00 00 48 c7 c6 90 0e 17 a0 48 c7 c7 20 f8 16 a0 31 c0 e8 > 8a 5d 08 e1 <0f> 0b 75 14 49 8b 7f 68 41 bd 92 ff ff ff e8 d4 e0 fc ff > e9 dc > [789881.911744] RIP [<ffffffffa016d1c9>] rbd_dev_header_info+0x5a9/0x940 [rbd] > [789881.919116] RSP <ffff8803f0c93d30> > [789881.922989] ---[ end trace 12b8d1c2ed74d6c1 ]--- > [789881.927971] BUG: unable to handle kernel paging request at ffffffffffffffd8 > [789881.935435] IP: [<ffffffff8112b821>] kthread_data+0x11/0x20 > [789881.941427] PGD 1e0a067 PUD 1e0c067 PMD 0 > [789881.946117] Oops: 0000 [#2] SMP > [789881.949591] Modules linked in: rbd libceph sg rpcsec_gss_krb5 > xt_nat xt_UDPLB(O) xt_multiport xt_addrtype iptable_mangle iptable_raw > iptable_nat nf_nat_ipv4 nf_nat ext4 jbd2 mbcache x86_pkg_temp_thermal > gkuart(O) usbserial ie31200_edac edac_core tpm_tis raid1 crc32c_intel > [789881.976900] CPU: 4 PID: 4111 Comm: kworker/u16:0 Tainted: G D > O 4.7.0-vanilla-ams-3 #1 > [789881.986280] Hardware name: Quanta T6BC-S1N/T6BC, BIOS T6BC2A01 03/26/2014 > [789881.993410] task: ffff88032069ea00 ti: ffff8803f0c90000 task.ti: > ffff8803f0c90000 > [789882.001411] RIP: 0010:[<ffffffff8112b821>] [<ffffffff8112b821>] > kthread_data+0x11/0x20 > [789882.010024] RSP: 0018:ffff8803f0c93a28 EFLAGS: 00010002 > [789882.015682] RAX: 0000000000000000 RBX: ffff88041fd17e80 RCX: > 0000000000000004 > [789882.023342] RDX: ffff88040f005000 RSI: ffff88032069ea00 RDI: > ffff88032069ea00 > [789882.030996] RBP: ffff8803f0c93a30 R08: 0000000000000000 R09: > 0000000000079800 > [789882.038645] R10: 0000000000000001 R11: 0000000000000001 R12: > 0000000000017e80 > [789882.046288] R13: 0000000000000000 R14: ffff88032069eec0 R15: > ffff88032069ea00 > [789882.053926] FS: 0000000000000000(0000) GS:ffff88041fd00000(0000) > knlGS:0000000000000000 > [789882.062524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [789882.068603] CR2: 0000000000000028 CR3: 0000000001e07000 CR4: > 00000000001406e0 > [789882.076261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [789882.083920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [789882.091577] Stack: > [789882.093935] ffffffff8112645e ffff8803f0c93a78 ffffffff8185ab3e > ffff88032069ea00 > [789882.102194] ffff8803f0c93a78 ffff8803f0c94000 ffff8803f0c93ad0 > ffff8803f0c936e8 > [789882.110438] ffff88040d5c8000 0000000000000000 ffff8803f0c93a90 > ffffffff8185aef5 > [789882.118689] Call Trace: > [789882.121484] [<ffffffff8112645e>] ? wq_worker_sleeping+0xe/0x90 > [789882.127752] [<ffffffff8185ab3e>] __schedule+0x36e/0x6f0 > [789882.133411] [<ffffffff8185aef5>] schedule+0x35/0x80 > [789882.138712] [<ffffffff81110ff9>] do_exit+0x739/0xb50 > [789882.144098] [<ffffffff8108833c>] oops_end+0x9c/0xd0 > [789882.149400] [<ffffffff810887ab>] die+0x4b/0x70 > [789882.154276] [<ffffffff81085b26>] do_trap+0xb6/0x150 > [789882.159583] [<ffffffff81085d87>] do_error_trap+0x77/0xe0 > [789882.165322] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd] > [789882.172446] [<ffffffff811d7a3d>] ? irq_work_queue+0x6d/0x80 > [789882.178441] [<ffffffff811575d4>] ? wake_up_klogd+0x34/0x40 > [789882.184363] [<ffffffff81157aa6>] ? console_unlock+0x4c6/0x510 > [789882.190532] [<ffffffff810863c0>] do_invalid_op+0x20/0x30 > [789882.196265] [<ffffffff8185fb6e>] invalid_op+0x1e/0x30 > [789882.201740] [<ffffffffa016d1c9>] ? rbd_dev_header_info+0x5a9/0x940 [rbd] > [789882.208866] [<ffffffff8113a91a>] ? update_curr+0x8a/0x110 > [789882.214694] [<ffffffff8113d968>] ? dequeue_task_fair+0x618/0x1150 > [789882.221225] [<ffffffffa016d591>] rbd_dev_refresh+0x31/0xf0 [rbd] > [789882.227662] [<ffffffffa016d719>] rbd_watch_cb+0x29/0xa0 [rbd] > [789882.233855] [<ffffffffa0138fdc>] do_watch_notify+0x4c/0x80 [libceph] > [789882.240647] [<ffffffff811258e9>] process_one_work+0x149/0x3c0 > [789882.246811] [<ffffffff81125bae>] worker_thread+0x4e/0x490 > [789882.252629] [<ffffffff81125b60>] ? process_one_work+0x3c0/0x3c0 > [789882.258969] [<ffffffff8112b1e9>] kthread+0xc9/0xe0 > [789882.264182] [<ffffffff8185e4ff>] ret_from_fork+0x1f/0x40 > [789882.269917] [<ffffffff8112b120>] ? kthread_create_on_node+0x170/0x170 > [789882.276784] Code: 02 00 00 00 e8 a1 fd ff ff 5d c3 0f 1f 44 00 00 > 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 60 04 00 00 55 > 48 89 e5 5d <48> 8b 40 d8 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 > 00 55 > [789882.301941] RIP [<ffffffff8112b821>] kthread_data+0x11/0x20 > [789882.308016] RSP <ffff8803f0c93a28> > [789882.311847] CR2: ffffffffffffffd8 > [789882.315505] ---[ end trace 12b8d1c2ed74d6c2 ]--- > [789882.320462] Fixing recursive fault but reboot is needed! That's the same one you've reported in the "ceph osd kernel divide error", right? I've filed http://tracker.ceph.com/issues/16963 and should get to it later this week. What did you mean by "no networking stack" in that thread? Thanks, Ilya -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
