On Fri, Apr 19, 2013 at 3:12 PM, Steven Presser <spresse1@xxxxxxxxxxx> wrote: > Hey all, > I've got a ceph cluster set up (0.56.4) on a custom centos image (base > centos 6, plus kernel 3.6.9) running as a Xen dom0. I'm seeing a lot of > messages like the ones at the bottom of this message. I'm entirely willing > to believe the hardware on these is going bad (it's donated hardware) but > have run stress tests on some of these and can't figure out what could be > failing. I'm likely to blame the Myricom fiber cards (old, I had to hack > the driver a bit to get them to run here...), but this looks like it doesn't > involve that. > > Any help or advice is appreciated. > > Thanks in advance, > Steve > > > 2013-04-19 17:47:50.360892 osd.0 [WRN] slow request 33.009444 seconds old, > received at 2013-04-19 17:47:17.351358: osd_op(client.6318.1:285339 > rb.0.1211.238e1f29.0000000000cd [write 3674112~520192] 2.2e1e015e RETRY) > currently waiting for ondisk So what these (generically) mean is that a request is taking a long time to get all the way through the OSD pipeline. This one in particular is an op that's being replayed, and "waiting for ondisk", which means it's waiting for a write to get committed to disk everywhere. What workload are you running and have you looked at your general system utilization? This could happen if you're just overloading your OSDs with too many writes, for instance. However, given the particular path you've hit, you might also want to check your network cards — the connection got reset between the OSD and the client, but the OSD remained healthy enough (in its own observation and that of its peers) to stay alive. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com > > This is (eventually) accompanied by a panic much like: > general protection fault: 0000 [#1] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 0 > Pid: 8478, comm: ceph-osd Tainted: P O 3.6.9-7.el6.x86_64 #1 To be > filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff811221f9>] [<ffffffff811221f9>] put_page+0x9/0x40 > RSP: e02b:ffff88000b247dc8 EFLAGS: 00010212 > RAX: 0000000000000110 RBX: 0000000000000012 RCX: 0000000000000010 > RDX: ffff8800517a2ec0 RSI: ffff880076616840 RDI: 483f76ecd761f21e > RBP: ffff88000b247dc8 R08: ffffea00004b78c8 R09: 0000000000000000 > R10: ffff880076454e80 R11: 0000000000000293 R12: ffff88006cf9cc80 > general protection fault: 0000 [#1] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 0 > Pid: 8478, comm: ceph-osd Tainted: P O 3.6.9-7.el6.x86_64 #1 To be > filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff811221f9>] [<ffffffff811221f9>] put_page+0x9/0x40 > RSP: e02b:ffff88000b247dc8 EFLAGS: 00010212 > RAX: 0000000000000110 RBX: 0000000000000012 RCX: 0000000000000010 > RDX: ffff8800517a2ec0 RSI: ffff880076616840 RDI: 483f76ecd761f21e > RBP: ffff88000b247dc8 R08: ffffea00004b78c8 R09: 0000000000000000 > R10: ffff880076454e80 R11: 0000000000000293 R12: ffff88006cf9cc80general > protection fault: 0000 [#1] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 0 > Pid: 8478, comm: ceph-osd Tainted: P O 3.6.9-7.el6.x86_64 #1 To be > filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff811221f9>] [<ffffffff811221f9>] put_page+0x9/0x40 > RSP: e02b:ffff88000b247dc8 EFLAGS: 00010212 > RAX: 0000000000000110 RBX: 0000000000000012 RCX: 0000000000000010 > RDX: ffff8800517a2ec0 RSI: ffff880076616840 RDI: 483f76ecd761f21e > RBP: ffff88000b247dc8 R08: ffffea00004b78c8 R09: 0000000000000000 > R10: ffff880076454e80 R11: 0000000000000293 R12: ffff88006cf9cc80 > R13: 000000000000d1b8 R14: ffff88006cf7e230 R15: ffff880061cfdd70 > FS: 00007ffc0d9e7700(0000) GS:ffff880076600000(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffffff600400 CR3: 0000000062175000 CR4: 0000000000000660 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process ceph-osd (pid: 8478, threadinfo ffff88000b246000, task > ffff880003ec7460) > Stack: > ffff88000b247de8 ffffffff8146cb03 ffff88006cf9cc80 0000000000000000 > ffff88000b247e08 ffffffff8146cb9e 0000000000000000 ffff88006cf7e1c0 > ffff88000b247e38 ffffffff814bd53e ffff88006cf7e1c0 ffff880061cfdd40 > Call Trace: > [<ffffffff8146cb03>] skb_release_data+0x73/0xf0 > [<ffffffff8146cb9e>] __kfree_skb+0x1e/0xa0 > [<ffffffff814bd53e>] tcp_close+0x8e/0x3d0 > [<ffffffff814e4c5e>] inet_release+0x5e/0x80 > [<ffffffff81461969>] sock_release+0x29/0x90 > [<ffffffff814619e7>] sock_close+0x17/0x30 > [<ffffffff8116e573>] __fput+0xb3/0x260 > [<ffffffff8116e78e>] ____fput+0xe/0x10 > [<ffffffff81073f7c>] task_work_run+0x6c/0x90 > [<ffffffff810149ac>] do_notify_resume+0x8c/0xa0 > [<ffffffff8116ad83>] ? filp_close+0x63/0x90 > [<ffffffff81562362>] int_signal+0x12/0x17 > Code: 66 90 e9 0f ff ff ff 89 c1 e9 2e ff ff ff f3 90 48 8b 02 a9 00 00 00 > 01 75 f4 eb 88 66 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 <66> f7 07 00 c0 > 75 17 f0 ff 4f 1c 0f 94 c0 84 c0 75 05 c9 c3 0f > RIP [<ffffffff811221f9>] put_page+0x9/0x40 > RSP <ffff88000b247dc8> > ---[ end trace a36b2ce7db9c7446 ]--- > general protection fault: 0000 [#2] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 1 > Pid: 6400, comm: ceph-osd Tainted: P D O 3.6.9-7.el6.x86_64 #1 To be > filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff811221f9>] [<ffffffff811221f9>] put_page+0x9/0x40 > RSP: e02b:ffff8800620b1dc8 EFLAGS: 00010212 > RAX: 0000000000000110 RBX: 0000000000000012 RCX: 0000000000000010 > RDX: ffff88004b8feec0 RSI: ffff880076716840 RDI: b90db395813cdde1 > RBP: ffff8800620b1dc8 R08: ffffea000017b1d8 R09: 0000000000000000 > R10: ffff880076454e80 R11: 0000000000000293 R12: ffff880067434ac0 > R13: 000000000000d1b8 R14: ffff88006e2ad7b0 R15: ffff88006ffc70b0 > FS: 00007ffc45680700(0000) GS:ffff880076700000(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffffff600400 CR3: 0000000062175000 CR4: 0000000000000660 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process ceph-osd (pid: 6400, threadinfo ffff8800620b0000, task > ffff88006e2a54e0) > Stack: > ffff8800620b1de8 ffffffff8146cb03 ffff880067434ac0 0000000000000000 > ffff8800620b1e08 ffffffff8146cb9e 0000000000000000 ffff88006e2ad740 > ffff8800620b1e38 ffffffff814bd53e ffff88006e2ad740 ffff88006ffc7080 > Call Trace: > [<ffffffff8146cb03>] skb_release_data+0x73/0xf0 > [<ffffffff8146cb9e>] __kfree_skb+0x1e/0xa0 > [<ffffffff814bd53e>] tcp_close+0x8e/0x3d0 > [<ffffffff814e4c5e>] inet_release+0x5e/0x80 > [<ffffffff81461969>] sock_release+0x29/0x90 > [<ffffffff814619e7>] sock_close+0x17/0x30 > [<ffffffff8116e573>] __fput+0xb3/0x260 > [<ffffffff8116e78e>] ____fput+0xe/0x10 > [<ffffffff81073f7c>] task_work_run+0x6c/0x90 > [<ffffffff810149ac>] do_notify_resume+0x8c/0xa0 > [<ffffffff8116ad83>] ? filp_close+0x63/0x90 > [<ffffffff81562362>] int_signal+0x12/0x17 > Code: 66 90 e9 0f ff ff ff 89 c1 e9 2e ff ff ff f3 90 48 8b 02 a9 00 00 00 > 01 75 f4 eb 88 66 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 <66> f7 07 00 c0 > 75 17 f0 ff 4f 1c 0f 94 c0 84 c0 75 05 c9 c3 0f > RIP [<ffffffff811221f9>] put_page+0x9/0x40 > RSP <ffff8800620b1dc8> > ---[ end trace a36b2ce7db9c7447 ]--- > > kernel BUG at net/ceph/osd_client.c:598! > invalid opcode: 0000 [#4] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 0 > Pid: 8556, comm: kworker/0:0 Tainted: P D O 3.6.9-7.el6.x86_64 #1 To > be filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff815320ea>] [<ffffffff815320ea>] > __kick_osd_requests+0x17a/0x1f0 > RSP: e02b:ffff88004e587d20 EFLAGS: 00010297 > RAX: ffff8800159fb820 RBX: ffff8800046f1450 RCX: 0000000000000000 > RDX: ffff8800159fb850 RSI: ffff880076600200 RDI: ffff880076600200 > RBP: ffff88004e587d60 R08: ffff88007660e420 R09: ffd8264126645403 > R10: 0000000000000000 R11: 0000000000000000 R12: ffff880062018740 > R13: ffff88006c9f8cb0 R14: ffff8800159fb800 R15: ffffffff8182b3a2 > FS: 00007f25814cc7c0(0000) GS:ffff880076600000(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: ffffffffff600400 CR3: 000000006d49d000 CR4: 0000000000000660 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/0:0 (pid: 8556, threadinfo ffff88004e586000, task > ffff880003fff560) > Stack: > ffff88004e587d40 ffff88006c9f8800 0000000000000000 ffff88006c9f8800 > ffff880062018740 ffff880062018750 ffff880062018798 ffffffff8182b3a2 > ffff88004e587d90 ffffffff815324fa ffff88006c9f8830 ffff88006c9f8860 > Call Trace: > [<ffffffff815324fa>] osd_reset+0x5a/0xb0 > [<ffffffff8152c0ab>] ceph_fault+0xfb/0x380 > [<ffffffff8152d690>] ? try_read+0x6a0/0x6a0 > [<ffffffff8152d84b>] con_work+0x1bb/0x2c0 > [<ffffffff810706d9>] process_one_work+0x179/0x4b0 > [<ffffffff81071dd6>] worker_thread+0x126/0x320 > [<ffffffff81071cb0>] ? manage_workers+0x190/0x190 > [<ffffffff8107766e>] kthread+0x9e/0xb0 > [<ffffffff81563144>] kernel_thread_helper+0x4/0x10 > [<ffffffff8155a0b8>] ? retint_restore_args+0x5/0x6 > [<ffffffff81563140>] ? gs_change+0x13/0x13 > Code: e7 e8 db f5 ff ff f6 05 56 b0 5a 00 04 75 20 48 8d 43 50 48 8b 53 50 > 49 39 c5 75 8d 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f c9 c3 <0f> 0b eb fe 48 > 8b 55 c8 49 8b 0e 48 c7 c6 18 35 83 81 48 c7 c7 > RIP [<ffffffff815320ea>] __kick_osd_requests+0x17a/0x1f0 > RSP <ffff88004e587d20> > ---[ end trace a36b2ce7db9c7449 ]--- > BUG: unable to handle kernel paging request at fffffffffffffff8 > IP: [<ffffffff810770a0>] kthread_data+0x10/0x20 > PGD 1a0d067 PUD 1a0e067 PMD 0 > Oops: 0000 [#5] SMP > Modules linked in: cbc ip6table_filter ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables > bridge stp llc xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev > xen_evtchn xenfs xen_privcmd btrfs zlib_deflate openafs(PO) mx_driver(PO) > mx_mcp(PO) autofs4 nfsv4 auth_rpcgss nfs fscache lockd sunrpc ipv6 tg3 ppdev > freq_table mperf pcspkr serio_raw k8temp edac_core edac_mce_amd i2c_amd756 > amd_rng i2c_amd8111 i2c_core parport_pc parport sg shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif floppy sata_sil pata_acpi ata_generic pata_amd dm_mirror > dm_region_hash dm_log dm_mod > CPU 0 > Pid: 8556, comm: kworker/0:0 Tainted: P D O 3.6.9-7.el6.x86_64 #1 To > be filled by O.E.M. To be filles by O.E.M./S2880 Thunder K8S > RIP: e030:[<ffffffff810770a0>] [<ffffffff810770a0>] kthread_data+0x10/0x20 > RSP: e02b:ffff88004e5879f8 EFLAGS: 00010082 > RAX: 0000000000000000 RBX: ffff8800766139c0 RCX: 0000000000000000 > RDX: ffffffff81d55aa0 RSI: 0000000000000000 RDI: ffff880003fff560 > RBP: ffff88004e5879f8 R08: ffff880003fff5d0 R09: 0000000000000001 > R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 > R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000006 > FS: 00007f272ffff700(0000) GS:ffff880076600000(0000) knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: fffffffffffffff8 CR3: 000000001589a000 CR4: 0000000000000660 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/0:0 (pid: 8556, threadinfo ffff88004e586000, task > ffff880003fff560) > Stack: > ffff88004e587a28 ffffffff8106fc41 ffff88004e587a28 ffff8800766139c0 > 0000000000000000 ffff880003fffb10 ffff88004e587ab8 ffffffff81558d03 > ffff88004e587fd8 00000000000139c0 ffff88004e586010 00000000000139c0 > Call Trace: > [<ffffffff8106fc41>] wq_worker_sleeping+0x21/0xa0 > [<ffffffff81558d03>] __schedule+0x593/0x700 > [<ffffffff81559199>] schedule+0x29/0x70 > [<ffffffff8105a4cd>] do_exit+0x2bd/0x470 > [<ffffffff8155aebb>] oops_end+0xab/0xf0 > [<ffffffff810175db>] die+0x5b/0x90 > [<ffffffff8155aa14>] do_trap+0xc4/0x160 > [<ffffffff810154f5>] do_invalid_op+0x95/0xb0 > [<ffffffff815320ea>] ? __kick_osd_requests+0x17a/0x1f0 > [<ffffffff8106fded>] ? __queue_work+0xfd/0x3d0 > [<ffffffff810df03d>] ? __call_rcu_core+0xbd/0x170 > [<ffffffff81562fbb>] invalid_op+0x1b/0x20 > [<ffffffff815320ea>] ? __kick_osd_requests+0x17a/0x1f0 > [<ffffffff815324fa>] osd_reset+0x5a/0xb0 > [<ffffffff8152c0ab>] ceph_fault+0xfb/0x380 > [<ffffffff8152d690>] ? try_read+0x6a0/0x6a0 > [<ffffffff8152d84b>] con_work+0x1bb/0x2c0 > [<ffffffff810706d9>] process_one_work+0x179/0x4b0 > [<ffffffff81071dd6>] worker_thread+0x126/0x320 > [<ffffffff81071cb0>] ? manage_workers+0x190/0x190 > [<ffffffff8107766e>] kthread+0x9e/0xb0 > [<ffffffff81563144>] kernel_thread_helper+0x4/0x10 > [<ffffffff8155a0b8>] ? retint_restore_args+0x5/0x6 > [<ffffffff81563140>] ? gs_change+0x13/0x13 > Code: 66 66 66 90 65 48 8b 04 25 40 c6 00 00 48 8b 80 58 05 00 00 8b 40 f0 > c9 c3 66 90 55 48 89 e5 66 66 66 66 90 48 8b 87 58 05 00 00 <48> 8b 40 f8 c9 > c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 > RIP [<ffffffff810770a0>] kthread_data+0x10/0x20 > RSP <ffff88004e5879f8> > CR2: fffffffffffffff8 > ---[ end trace a36b2ce7db9c744a ]--- > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |