David, I observed a new when un-registering the inode cookie. I haven't seen this before although I've been using this kernel for a while. On unregister it fails an assert of 0 > 0 of n_active. This is new machine (different) hardware. I suspect this is a some kind of race bug due to the error message (and possibly different hardware). Do you have some pointers where to look? FS-Cache: Assertion failed 0 > 0 is false ------------[ cut here ]------------ kernel BUG at fs/fscache/cookie.c:519! invalid opcode: 0000 [#1] SMP Modules linked in: ceph libceph libcrc32c cachefiles raid0 auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc ixgbevf CPU: 7 PID: 27157 Comm: petabucket Not tainted 3.14.0-virtual #72 Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 task: ffff880f0ac397f0 ti: ffff880f0ac5c000 task.ti: ffff880f0ac5c000 RIP: 0010:[<ffffffffa005584b>] [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP: 0018:ffff880f0ac5dab8 EFLAGS: 00010292 RAX: 000000000000000e RBX: ffff880f026222d0 RCX: 0000000000000000 RDX: ffff880f520efe50 RSI: ffff880f520ee318 RDI: 0000000000000246 RBP: ffff880f0ac5dad8 R08: ffffffff81acf658 R09: 0000000000000000 R10: 00000000000002bc R11: 00000000000002bb R12: 0000000000000000 R13: ffff880eca794320 R14: ffffffffa01db120 R15: ffff880f0bdbea20 FS: 00007fdbf03ff700(0000) GS:ffff880f520e0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fdbe500a940 CR3: 0000000f0bca9000 CR4: 00000000001407e0 Stack: ffff880f026222d0 0000000000000000 ffff880eca794320 ffffffffa01db120 ffff880f0ac5db08 ffffffffa0056060 ffff880eca794780 ffff880eca794320 ffff880f026222d0 ffff880eca794780 ffff880f0ac5db28 ffffffffa01da4ee Call Trace: [<ffffffffa0056060>] __fscache_relinquish_cookie+0x50/0x220 [fscache] [<ffffffffa01da4ee>] ceph_fscache_unregister_inode_cookie+0x3e/0x50 [ceph] [<ffffffffa01b21d3>] ceph_destroy_inode+0x33/0x200 [ceph] [<ffffffff811d4e8e>] ? __fsnotify_inode_delete+0xe/0x10 [<ffffffff811ae8cc>] destroy_inode+0x3c/0x70 [<ffffffff811aea19>] evict+0x119/0x1b0 [<ffffffff811af253>] iput+0x103/0x190 [<ffffffff811aae70>] dentry_kill+0x210/0x260 [<ffffffff811ab214>] dput+0x64/0x100 [<ffffffff8119f3fd>] lookup_fast+0x29d/0x2e0 [<ffffffff811a341f>] do_last+0xbf/0xf40 [<ffffffff8119fbb8>] ? inode_permission+0x18/0x50 [<ffffffff811a022e>] ? link_path_walk+0x22e/0x8a0 [<ffffffff811a4364>] path_openat+0xc4/0x630 [<ffffffff811a4df3>] do_filp_open+0x43/0xa0 [<ffffffff811b1175>] ? __alloc_fd+0x45/0x120 [<ffffffff811939c2>] do_sys_open+0x132/0x220 [<ffffffff81193ace>] SyS_open+0x1e/0x20 [<ffffffff815a142d>] system_call_fastpath+0x1a/0x1f Code: 31 c0 e8 a5 92 53 e1 48 c7 c7 a8 09 06 a0 31 c0 e8 97 92 53 e1 8b 73 08 31 d2 48 c7 c7 66 0a 06 a0 31 c0 48 63 f6 e8 81 92 53 e1 <0f> 0b 65 48 8b 34 25 80 c7 00 00 48 89 f9 48 81 c6 60 05 00 00 RIP [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP <ffff880f0ac5dab8> ---[ end trace 94bafbdfe9185924 ]--- FS-Cache: Assertion failed 0 > 0 is false ------------[ cut here ]------------ kernel BUG at fs/fscache/cookie.c:519! invalid opcode: 0000 [#2] SMP Modules linked in: ceph libceph libcrc32c cachefiles raid0 auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc ixgbevf CPU: 6 PID: 29394 Comm: petabucket Tainted: G D 3.14.0-virtual #72 Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 task: ffff880effe7afe0 ti: ffff880efff06000 task.ti: ffff880efff06000 RIP: 0010:[<ffffffffa005584b>] [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP: 0018:ffff880efff07ab8 EFLAGS: 00010292 RAX: 000000000000000e RBX: ffff880f07e70230 RCX: 0000000000000006 RDX: 0000000000000007 RSI: 0000000000000007 RDI: 0000000000000246 RBP: ffff880efff07ad8 R08: ffffffff81acf658 R09: 0000000000000000 R10: 00000000000002ec R11: 00000000000002eb R12: 0000000000000000 R13: ffff8800e82b3d88 R14: ffffffffa01db120 R15: ffff880f0bdbea20 FS: 00007fdbe47ff700(0000) GS:ffff880f520c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fdbe5c47000 CR3: 0000000f0bca9000 CR4: 00000000001407e0 Stack: ffff880f07e70230 0000000000000000 ffff8800e82b3d88 ffffffffa01db120 ffff880efff07b08 ffffffffa0056060 ffff8800e82b41e8 ffff8800e82b3d88 ffff880f07e70230 ffff8800e82b41e8 ffff880efff07b28 ffffffffa01da4ee Call Trace: [<ffffffffa0056060>] __fscache_relinquish_cookie+0x50/0x220 [fscache] [<ffffffffa01da4ee>] ceph_fscache_unregister_inode_cookie+0x3e/0x50 [ceph] [<ffffffffa01b21d3>] ceph_destroy_inode+0x33/0x200 [ceph] [<ffffffff811d4e8e>] ? __fsnotify_inode_delete+0xe/0x10 [<ffffffff811ae8cc>] destroy_inode+0x3c/0x70 [<ffffffff811aea19>] evict+0x119/0x1b0 [<ffffffff811af253>] iput+0x103/0x190 [<ffffffff811aae70>] dentry_kill+0x210/0x260 [<ffffffff811ab214>] dput+0x64/0x100 [<ffffffff8119f3fd>] lookup_fast+0x29d/0x2e0 [<ffffffff811a341f>] do_last+0xbf/0xf40 [<ffffffff8119fbb8>] ? inode_permission+0x18/0x50 [<ffffffff811a022e>] ? link_path_walk+0x22e/0x8a0 [<ffffffff811a4364>] path_openat+0xc4/0x630 [<ffffffffa01d202b>] ? ceph_mdsc_release_request+0x16b/0x1a0 [ceph] [<ffffffff811a4df3>] do_filp_open+0x43/0xa0 [<ffffffff811b1175>] ? __alloc_fd+0x45/0x120 [<ffffffff811939c2>] do_sys_open+0x132/0x220 [<ffffffff81193ace>] SyS_open+0x1e/0x20 [<ffffffff815a142d>] system_call_fastpath+0x1a/0x1f Code: 31 c0 e8 a5 92 53 e1 48 c7 c7 a8 09 06 a0 31 c0 e8 97 92 53 e1 8b 73 08 31 d2 48 c7 c7 66 0a 06 a0 31 c0 48 63 f6 e8 81 92 53 e1 <0f> 0b 65 48 8b 34 25 80 c7 00 00 48 89 f9 48 81 c6 60 05 00 00 RIP [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP <ffff880efff07ab8> ---[ end trace 94bafbdfe9185925 ]--- init: petabucket main process (22513) killed by KILL signal FS-Cache: Assertion failed fffffffffffffffc > 0 is false ------------[ cut here ]------------ kernel BUG at fs/fscache/cookie.c:519! invalid opcode: 0000 [#3] SMP Modules linked in: ceph libceph libcrc32c cachefiles raid0 auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc ixgbevf CPU: 6 PID: 2891 Comm: petabucket Tainted: G D 3.14.0-virtual #72 Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 task: ffff880f026797f0 ti: ffff880effe66000 task.ti: ffff880effe66000 RIP: 0010:[<ffffffffa005584b>] [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP: 0018:ffff880effe67ad8 EFLAGS: 00010292 RAX: 000000000000001d RBX: ffff880f07e70aa0 RCX: 0000000000000000 RDX: ffff880f520cfe50 RSI: ffff880f520ce318 RDI: 0000000000000246 RBP: ffff880effe67af8 R08: ffffffff81acf658 R09: 0000000000000000 R10: 0000000000000322 R11: 0000000000000321 R12: 0000000000000000 R13: ffff880eca855f18 R14: ffffffffa01db120 R15: ffff880f0bdbea20 FS: 00007ffa58fff700(0000) GS:ffff880f520c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffa5e44b000 CR3: 0000000f0dd61000 CR4: 00000000001407e0 Stack: ffff880f07e70aa0 0000000000000000 ffff880eca855f18 ffffffffa01db120 ffff880effe67b28 ffffffffa0056060 ffff880eca856378 ffff880eca855f18 ffff880f07e70aa0 ffff880eca856378 ffff880effe67b48 ffffffffa01da4ee Call Trace: [<ffffffffa0056060>] __fscache_relinquish_cookie+0x50/0x220 [fscache] [<ffffffffa01da4ee>] ceph_fscache_unregister_inode_cookie+0x3e/0x50 [ceph] [<ffffffffa01b21d3>] ceph_destroy_inode+0x33/0x200 [ceph] [<ffffffff811d4e8e>] ? __fsnotify_inode_delete+0xe/0x10 [<ffffffff811ae8cc>] destroy_inode+0x3c/0x70 [<ffffffff811aea19>] evict+0x119/0x1b0 [<ffffffff811af253>] iput+0x103/0x190 [<ffffffff811aae70>] dentry_kill+0x210/0x260 [<ffffffff811ab214>] dput+0x64/0x100 [<ffffffff8119f3fd>] lookup_fast+0x29d/0x2e0 [<ffffffff811a09d4>] path_lookupat+0x134/0x6f0 [<ffffffff8113f6c7>] ? release_pages+0x77/0x200 [<ffffffff811a0fc4>] filename_lookup+0x34/0xc0 [<ffffffff811a4cd9>] user_path_at_empty+0x59/0xa0 [<ffffffff81160caa>] ? unmap_region+0xea/0x110 [<ffffffff811a4d31>] user_path_at+0x11/0x20 [<ffffffff811996b1>] vfs_fstatat+0x51/0xb0 [<ffffffff811997db>] vfs_stat+0x1b/0x20 [<ffffffff811997f5>] SYSC_newstat+0x15/0x30 [<ffffffff810c10fe>] ? getnstimeofday+0xe/0x30 [<ffffffff810c118a>] ? do_gettimeofday+0x1a/0x50 [<ffffffff8106780a>] ? SyS_gettimeofday+0x3a/0x80 [<ffffffff8119994e>] SyS_newstat+0xe/0x10 [<ffffffff815a142d>] system_call_fastpath+0x1a/0x1f Code: 31 c0 e8 a5 92 53 e1 48 c7 c7 a8 09 06 a0 31 c0 e8 97 92 53 e1 8b 73 08 31 d2 48 c7 c7 66 0a 06 a0 31 c0 48 63 f6 e8 81 92 53 e1 <0f> 0b 65 48 8b 34 25 80 c7 00 00 48 89 f9 48 81 c6 60 05 00 00 RIP [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP <ffff880effe67ad8> ---[ end trace 94bafbdfe9185926 ]--- FS-Cache: Assertion failed fffffffffffffff2 > 0 is false ------------[ cut here ]------------ kernel BUG at fs/fscache/cookie.c:519! invalid opcode: 0000 [#4] SMP Modules linked in: ceph libceph libcrc32c cachefiles raid0 auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc ixgbevf CPU: 7 PID: 2892 Comm: petabucket Tainted: G D 3.14.0-virtual #72 Hardware name: Xen HVM domU, BIOS 4.2.amazon 01/24/2014 task: ffff880f0267afe0 ti: ffff880f0aa2a000 task.ti: ffff880f0aa2a000 RIP: 0010:[<ffffffffa005584b>] [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP: 0018:ffff880f0aa2bad8 EFLAGS: 00010292 RAX: 000000000000001d RBX: ffff880f02622aa0 RCX: 0000000000000000 RDX: ffff880f520efe50 RSI: ffff880f520ee318 RDI: 0000000000000246 RBP: ffff880f0aa2baf8 R08: ffffffff81acf658 R09: 0000000000000000 R10: 0000000000000357 R11: 0000000000000356 R12: 0000000000000000 R13: ffff880eca9eefe0 R14: ffffffffa01db120 R15: ffff880f0bdbea20 FS: 00007ffa587fe700(0000) GS:ffff880f520e0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffa5dc4a000 CR3: 0000000f0dd61000 CR4: 00000000001407e0 Stack: ffff880f02622aa0 0000000000000000 ffff880eca9eefe0 ffffffffa01db120 ffff880f0aa2bb28 ffffffffa0056060 ffff880eca9ef440 ffff880eca9eefe0 ffff880f02622aa0 ffff880eca9ef440 ffff880f0aa2bb48 ffffffffa01da4ee Call Trace: [<ffffffffa0056060>] __fscache_relinquish_cookie+0x50/0x220 [fscache] [<ffffffffa01da4ee>] ceph_fscache_unregister_inode_cookie+0x3e/0x50 [ceph] [<ffffffffa01b21d3>] ceph_destroy_inode+0x33/0x200 [ceph] [<ffffffff811d4e8e>] ? __fsnotify_inode_delete+0xe/0x10 [<ffffffff811ae8cc>] destroy_inode+0x3c/0x70 [<ffffffff811aea19>] evict+0x119/0x1b0 [<ffffffff811af253>] iput+0x103/0x190 [<ffffffff811aae70>] dentry_kill+0x210/0x260 [<ffffffff811ab214>] dput+0x64/0x100 [<ffffffff8119f3fd>] lookup_fast+0x29d/0x2e0 [<ffffffff811a09d4>] path_lookupat+0x134/0x6f0 [<ffffffff8113f6c7>] ? release_pages+0x77/0x200 [<ffffffff811a0fc4>] filename_lookup+0x34/0xc0 [<ffffffff811a4cd9>] user_path_at_empty+0x59/0xa0 [<ffffffff81160caa>] ? unmap_region+0xea/0x110 [<ffffffff811a4d31>] user_path_at+0x11/0x20 [<ffffffff811996b1>] vfs_fstatat+0x51/0xb0 [<ffffffff811997db>] vfs_stat+0x1b/0x20 [<ffffffff811997f5>] SYSC_newstat+0x15/0x30 [<ffffffff810c10fe>] ? getnstimeofday+0xe/0x30 [<ffffffff810c118a>] ? do_gettimeofday+0x1a/0x50 [<ffffffff8106780a>] ? SyS_gettimeofday+0x3a/0x80 [<ffffffff8119994e>] SyS_newstat+0xe/0x10 [<ffffffff815a142d>] system_call_fastpath+0x1a/0x1f Code: 31 c0 e8 a5 92 53 e1 48 c7 c7 a8 09 06 a0 31 c0 e8 97 92 53 e1 8b 73 08 31 d2 48 c7 c7 66 0a 06 a0 31 c0 48 63 f6 e8 81 92 53 e1 <0f> 0b 65 48 8b 34 25 80 c7 00 00 48 89 f9 48 81 c6 60 05 00 00 RIP [<ffffffffa005584b>] __fscache_disable_cookie+0x1db/0x210 [fscache] RSP <ffff880f0aa2bad8> ---[ end trace 94bafbdfe9185927 ]--- -- Milosz Tanski CTO 10 East 53rd Street, 37th floor New York, NY 10022 p: 646-253-9055 e: milosz@xxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
