Hey guys, I've ran into this issue with a soft lookup in the client when using the Ceph filesystem (trace bellow). I've ran into this twice so far on my cluster (separated by days) when my application is doing work (opening a new database segment). It's entirely possible this might be self inflicted with the fscache code I've been working on. I'm investigating what I could have done to cause it, but in the mean time I'd like to know if this looks familiar to any other bug? Thanks, - Milosz [2938098.374766] BUG: soft lockup - CPU#1 stuck for 23s! [petabucket:2257] [2938098.374778] Modules linked in: ceph libceph cachefiles ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 microcode auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc raid10 raid456 async_pq async_xor async_memcpy async_raid6_recov async_tx raid1 multipath linear btrfs raid6_pq lzo_compress raid0 xor zlib_deflate lib crc32c [2938098.374803] CPU: 1 PID: 2257 Comm: petabucket Not tainted 3.10.0-rc6-virtual #9 [2938098.374805] task: ffff880ecd68c4d0 ti: ffff880ecff2e000 task.ti: ffff880ecff2e000 [2938098.374807] RIP: e030:[<ffffffff81553c42>] [<ffffffff81553c42>] _raw_spin_lock+0x22/0x30 [2938098.374816] RSP: e02b:ffff880ecff2fd88 EFLAGS: 00000202 [2938098.374817] RAX: 0000000000000045 RBX: ffff880eaf68a530 RCX: 000000000001518e [2938098.374819] RDX: 0000000000000046 RSI: 0000000000000001 RDI: ffff880eaf68a540 [2938098.374820] RBP: ffff880ecff2fd88 R08: 000000000001a490 R09: ffffea0039d01400 [2938098.374822] R10: ffffffffa02bcc82 R11: 0000000000000000 R12: ffff880ed0254a70 [2938098.374823] R13: 0000000000000000 R14: 0000000000000001 R15: ffff880ecb3dbc00 [2938098.374828] FS: 00007f1231eba700(0000) GS:ffff880f1b420000(0000) knlGS:0000000000000000 [2938098.374830] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [2938098.374832] CR2: 00007f1241bf0000 CR3: 0000000ecc3c2000 CR4: 0000000000002660 [2938098.374833] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [2938098.374835] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [2938098.374836] Stack: [2938098.374837] ffff880ecff2fdb8 ffffffffa02ee92c ffff880ed0254800 ffff880ed0254a70 [2938098.374840] 0000000000000000 0000000000000155 ffff880ecff2fdd8 ffffffffa02f40eb [2938098.374843] ffff880eaf68a848 ffff880ed0254800 ffff880ecff2fe18 ffffffffa02e410c [2938098.374847] Call Trace: [2938098.374856] [<ffffffffa02ee92c>] ceph_put_cap_refs+0x2c/0x1c0 [ceph] [2938098.374861] [<ffffffffa02f40eb>] ceph_mdsc_release_request+0x8b/0x190 [ceph] [2938098.374865] [<ffffffffa02e410c>] ceph_do_getattr+0xfc/0x110 [ceph] [2938098.374869] [<ffffffffa02e4144>] ceph_getattr+0x24/0x100 [ceph] [2938098.374874] [<ffffffff81175a4d>] vfs_getattr+0x4d/0x80 [2938098.374876] [<ffffffff81175c9d>] vfs_fstat+0x3d/0x70 [2938098.374879] [<ffffffff81175ce5>] SYSC_newfstat+0x15/0x30 [2938098.374883] [<ffffffff8117beab>] ? putname+0x2b/0x40 [2938098.374888] [<ffffffff8116fc14>] ? do_sys_open+0x174/0x1e0 [2938098.374890] [<ffffffff81175d9e>] SyS_newfstat+0xe/0x10 [2938098.374895] [<ffffffff8155c559>] system_call_fastpath+0x16/0x1b [2938098.374896] Code: ff 48 89 d0 5d c3 0f 1f 00 66 66 66 66 90 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 0f b6 d4 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 <0f> b6 07 38 c2 75 f7 5d c3 0f 1f 44 00 00 66 66 66 66 90 55 48 -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
