Hi, is there any update regarding this bug? I can easily reproduce this issue on our cluster with the following scenario: - Start a few hundred processes on different nodes, each process writing slowly some text into its own output file - Call: watch -n1 'grep mycustomerrorstring *.out' - Hit CTRL+C (crashes the machine not always, but on a regular basis) We are using a 4.4.25 kernel with some additional ceph patches borrowed from newer kernel releases. Thanks, Markus -----Original Message----- From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Nikolay Borisov Sent: Montag, 10. Oktober 2016 12:36 To: Ilya Dryomov <idryomov@xxxxxxxxx> Cc: Yan, Zheng <zyan@xxxxxxxxxx>; ceph-users <ceph-users@xxxxxxxxxxxxxx> Subject: Re: Crash in ceph_read_iter->__free_pages due to null page On 10/10/2016 12:22 PM, Ilya Dryomov wrote: > On Fri, Oct 7, 2016 at 1:40 PM, Nikolay Borisov <kernel@xxxxxxxx> wrote: >> Hello, >> >> I've encountered yet another cephfs crash: >> >> [990188.822271] BUG: unable to handle kernel NULL pointer dereference >> at 000000000000001c [990188.822790] IP: [<ffffffff81130515>] >> __free_pages+0x5/0x30 [990188.823090] PGD 180dd8f067 PUD 1bf2722067 >> PMD 0 [990188.823506] Oops: 0002 [#1] SMP >> [990188.831274] CPU: 25 PID: 18418 Comm: php-fpm Tainted: G O 4.4.20-clouder2 #6 >> [990188.831650] Hardware name: Supermicro X10DRi/X10DRi, BIOS 2.0 >> 12/28/2015 [990188.831876] task: ffff8822a3b7b700 ti: >> ffff88022427c000 task.ti: ffff88022427c000 [990188.832249] RIP: >> 0010:[<ffffffff81130515>] [<ffffffff81130515>] __free_pages+0x5/0x30 >> [990188.832691] RSP: 0000:ffff88022427fda8 EFLAGS: 00010246 >> [990188.832914] RAX: 00000000fffffe00 RBX: 0000000000000f3d RCX: >> 00000000c0000100 [990188.833292] RDX: 00000000000047f2 RSI: >> 0000000000000000 RDI: 0000000000000000 [990188.833670] RBP: >> ffff88022427fe50 R08: ffff88022427c000 R09: 00038459d3aa3ee4 >> [990188.834049] R10: 000000013b00e4b8 R11: 0000000000000000 R12: >> 0000000000000000 [990188.834429] R13: ffff8802c5189f88 R14: >> ffff881091270ca8 R15: ffff88022427fe70 [990188.838820] FS: >> 00007fc8ff5cb7c0(0000) GS:ffff881fffba0000(0000) knlGS:0000000000000000 [990188.839197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [990188.839420] CR2: 000000000000001c CR3: 0000000405f7e000 CR4: 00000000001406e0 [990188.839797] Stack: >> [990188.840013] ffffffffa044a1bc ffff880600000000 0000000000000000 >> ffff88022427fe70 [990188.840639] ffff8802c5189f88 ffff88189297b6a0 >> ffffffff00000f3d ffff8810fffffe00 [990188.841263] ffff88022427fe98 >> 00000000ffffffff 0000000000002000 ffff8802c5189c20 [990188.841886] Call Trace: >> [990188.842115] [<ffffffffa044a1bc>] ? ceph_read_iter+0x19c/0x5f0 >> [ceph] [990188.842345] [<ffffffff81198c27>] __vfs_read+0xa7/0xd0 >> [990188.842568] [<ffffffff81199216>] vfs_read+0x86/0x130 >> [990188.842792] [<ffffffff81199fb6>] SyS_read+0x46/0xa0 >> [990188.843018] [<ffffffff81614f5b>] >> entry_SYSCALL_64_fastpath+0x16/0x6e >> [990188.843243] Code: e2 48 89 de ff d1 49 8b 0f 48 85 c9 75 e8 65 ff >> 0d 99 a7 ed 7e eb 85 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f >> 1f 44 00 00 <f0> ff 4f 1c 74 01 c3 55 85 f6 48 89 e5 74 07 e8 f7 f5 >> ff ff 5d [990188.847887] RIP [<ffffffff81130515>] >> __free_pages+0x5/0x30 [990188.848183] RSP <ffff88022427fda8> >> [990188.848404] CR2: 000000000000001c >> >> The problem is that page(%RDI) being passed to __free_pages is NULL. >> Also retry_op is CHECK_EOF(1), so the page allocation didn't execute >> which leads to the null page. statret is : fffffe00 which seems to be -ERESTARTSYS. > > Looks like this one exists upsteam - -ERESTARTSYS is returned from > __ceph_do_getattr() if the process is killed while waiting for the > reply from the MDS. At first sight it's just a busted error path, but > it could use more testing. Zheng? Checking the thread_info struct of the task in question it does have TIF_SIGPENDING set and indeed the crash's "sig" command (if I'm reading correctly the output" indicates that signal 15 (SIGTERM) is pending: SHARED_PENDING SIGNAL: 0000000000004000 SIGQUEUE: SIG SIGINFO 15 ffff8801439a5d78 > > Thanks, > > Ilya > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |