On Mon, Oct 3, 2016 at 1:19 PM, Nikolay Borisov <kernel@xxxxxxxx> wrote: > Hello, > > I've been investigating the following crash with cephfs: > > [8734559.785146] general protection fault: 0000 [#1] SMP > [8734559.791921] ioatdma shpchp ipmi_devintf ipmi_si ipmi_msghandler tcp_scalable ib_qib dca ib_mad ib_core ib_addr ipv6 [last unloaded: stat_faker_4410clouder4] > [8734559.793307] CPU: 31 PID: 1917 Comm: rsync Tainted: G W O 4.4.10-clouder4 #1 > [8734559.793686] Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1a 10/16/2015 > [8734559.793920] task: ffff883f3defc4c0 ti: ffff8821ef2e0000 task.ti: ffff8821ef2e0000 > [8734559.794306] RIP: 0010:[<ffffffff813134d0>] [<ffffffff813134d0>] lockref_get_not_dead+0x10/0xa0 > [8734559.794754] RSP: 0018:ffff8821ef2e3c28 EFLAGS: 00010296 > [8734559.794981] RAX: ffff881621afe000 RBX: 7261666153203689 RCX: 0000000000000007 > [8734559.795364] RDX: 0000000000000189 RSI: ffff8821ef2e3c38 RDI: 7261666153203689 > [8734559.795742] RBP: ffff8821ef2e3c68 R08: 0000000000000002 R09: 000000000000077d > [8734559.796130] R10: ffffea005886bf80 R11: ffff882cb6fe4e00 R12: 0000000000005c48 > [8734559.796511] R13: ffff8821ef2e3ef8 R14: 0000000000000000 R15: ffff88015aabcdd8 > [8734559.796892] FS: 00007fbed7c5e700(0000) GS:ffff881fffc60000(0000) knlGS:0000000000000000 > [8734559.797276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [8734559.797507] CR2: ffffffffff600400 CR3: 000000085bd95000 CR4: 00000000001406e0 > [8734559.797886] Stack: > [8734559.798110] ffff8821ef2e3c58 0000000000000000 ffff8821ef2e3c68 ffffffff8107b082 > [8734559.798747] 0000000000000000 ffffea005886bf80 0000000000005c48 7261666153203631 > [8734559.799388] ffff8821ef2e3d78 ffffffffa04ed6ae ffff8821ef2e3e58 ffff8821ef2e3d08 > [8734559.800032] Call Trace: > [8734559.800260] [<ffffffff8107b082>] ? __might_sleep+0x52/0x90 > [8734559.800496] [<ffffffffa04ed6ae>] __dcache_readdir+0x21e/0x480 [ceph] > [8734559.800727] [<ffffffff811ad482>] ? path_put+0x22/0x30 > [8734559.800957] [<ffffffffa04f67b8>] ? __touch_cap+0x28/0x90 [ceph] > [8734559.801195] [<ffffffffa04f6965>] ? ceph_cap_string+0xe5/0x100 [ceph] > [8734559.801432] [<ffffffffa04f6bb1>] ? __ceph_caps_issued_mask+0x141/0x150 [ceph] > [8734559.801819] [<ffffffffa04ee23a>] ceph_readdir+0x6ea/0x7d0 [ceph] > [8734559.802060] [<ffffffff8115e56a>] ? __might_fault+0x3a/0x50 > [8734559.802290] [<ffffffff811a87fa>] ? cp_new_stat+0x15a/0x180 > [8734559.802521] [<ffffffff8107b082>] ? __might_sleep+0x52/0x90 > [8734559.802751] [<ffffffff811b5b7e>] iterate_dir+0xae/0x130 > [8734559.802981] [<ffffffff811b5d90>] SyS_getdents+0x90/0x110 > [8734559.803216] [<ffffffff811b5ea0>] ? SyS_old_readdir+0x90/0x90 > [8734559.803445] [<ffffffff81639617>] entry_SYSCALL_64_fastpath+0x12/0x6a > [8734559.803673] Code: e8 56 5e 32 00 ff 43 04 c6 03 00 65 ff 0d 5d 77 cf 7e eb d2 0f 1f 80 00 00 00 00 55 48 89 e5 53 48 8d 75 d0 48 83 ec 38 48 89 fb <48> 8b 17 48 8d 7d e0 89 55 e0 48 89 55 c0 8b 45 e0 89 45 d0 85 > [8734559.808422] RIP [<ffffffff813134d0>] lockref_get_not_dead+0x10/0xa0 > [8734559.808721] RSP <ffff8821ef2e3c28> > > So the faulting instruction is (%rdi),%rdx, looking at the register > dump RDI clearly has a bogus value. I started backtracking from there > to acquire more context e.g. ge the state of the dir's ceph_inode_info > as well as the ceph_readdir_cache_control and here is what I found: > > 1. The dentry representing the dir which is being passed to __dcache_readdir: > http://sprunge.us/bAQH - the filename is rather strange, searching among the files > in the ceph mount point I couldn't find this file. Also, here is the state of the > ceph_inode_info: http://sprunge.us/AYRI > > crash> struct ceph_readdir_cache_control ffff8821ef2e3ce8 > struct ceph_readdir_cache_control { > page = 0xffffea005886bf80, > dentries = 0xffff881621afe000, > index = 2953 > } > > > According to the state of the ceph_inoide_info this means that > ceph_dir_is_complete_ordered would return true and the second condition > should also be true since ptr_pos is held in r12 and the dir size is 26496. > So the dentry being passed should be the 2953 % 512 = 393 in the cache_ctl.dentries array. > Unfortunately my crashdump excldues the page cache pages and I cannot really see > what are the contents of the dentries array. > > Could you provide any info on how to further debug this Zheng will know more, but this may have been fixed by https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=af5e5eb574776cdf1b756a27cc437bff257e22fe https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=a3d714c33632ef6bfdfaacc74ae6ba297b4c5820 in 4.6. Thanks, Ilya _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |