On Fri, Apr 4, 2014 at 4:25 PM, Hannes Landeholm <hannes@xxxxxxxxxxxxxx> wrote: > On Fri, Apr 4, 2014 at 1:08 PM, Ilya Dryomov <ilya.dryomov@xxxxxxxxxxx> wrote: >> On Wed, Apr 2, 2014 at 12:18 AM, Hannes Landeholm <hannes@xxxxxxxxxxxxxx> wrote: >>> Hello, >>> >>> We're running a couple of Arch Linux servers of version 3.13.5-1 in >>> production and suddenly one of them had a strange problem after >>> running for a few days. One process (pid 319) was running with a few >>> threads, one of those threads (pid 322) was eating 100% cpu. I assumed >>> it was stuck in an infinite loop (this was our own software so I >>> assumed we had a bug) so I sent a SIGKILL to 319 which caused all >>> other threads to exit and it turning into a zombie, but thread 322 was >>> still running. After trying to stop some other services and failing I >>> realized that sending any signals to any process now didn't work at >>> all in the system. >>> >>> This was the process stack output: >>> >>> $ cat /proc/319/stack >>> [<ffffffff810642fa>] do_exit+0x73a/0xa80 >>> [<ffffffff810646bf>] do_group_exit+0x3f/0xa0 >>> [<ffffffff81073295>] get_signal_to_deliver+0x295/0x5f0 >>> [<ffffffff810144a8>] do_signal+0x48/0x950 >>> [<ffffffff81014e18>] do_notify_resume+0x68/0xa0 >>> [<ffffffff8152326a>] int_signal+0x12/0x17 >>> [<ffffffffffffffff>] 0xffffffffffffffff >>> $ cat /proc/319/task/322/stack >>> [<ffffffff8151c11a>] error_exit+0x2a/0x60 >>> [<ffffffffffffffff>] 0xffffffffffffffff >>> >>> We're using ceph + rbd and this happened right after doing a rbd >>> mapping (mounting it) or during the mapping itself, so we suspected >>> rbd. >>> >>> A few days later (today) we had a server crash in another server, same >>> version+distro and it had also just been running a few days as well. >>> After starting it again we found the following in the system log: >>> >>> hostname kernel: BUG: unable to handle kernel paging request at ffff87fff75ad450 >>> hostname kernel: IP: [<ffffffffa018c196>] rbd_img_request_fill+0x126/0x930 [rbd] >> >> Can you try gdb'ing that exact rbd.ko and >> >> (gdb) list *rbd_img_request_fill+0x126 >> >> Also, the entire stack trace pertaining to rbd_img_request_fill would >> help. >> >> Thanks, >> >> Ilya > > Sorry, rbd was not built with any symbols and there was no other > output in the journal. > > We're building a debug version of Linux 3.14 (with symbols) now that > is currently being tested and we hope that we can roll it out today. I > saw that at least one commit (638c323 rbd: drop an unsafe assertion) > looks like something that could fix a crash that has been happening > regularly on one of our development servers: > http://i.imgur.com/pEKsmql.jpg (rbd_img_obj_callback in the stack > trace). It might be related to the production crash we had. Yeah, it most probably is. However, a bogus dereference in rbd_img_request_fill is something I haven't seen before. If you have that rbd.ko lying around, send it along, just in case. Thanks, Ilya -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
