Ilya, so, what is the best action plan now? should I continue using the kernel that you've sent me? I am running production infrastructure and not sure if this is the right way forward.
Do you have a patch by any chance against the LTS kernel that I can use to recompile the ceph module?
Thanks
Do you have a patch by any chance against the LTS kernel that I can use to recompile the ceph module?
Thanks
From: "Ilya Dryomov" <ilya.dryomov@xxxxxxxxxxx>
To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Sent: Saturday, 29 November, 2014 8:45:54 AM
Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
On Sat, Nov 29, 2014 at 2:13 AM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote:
> Ilya, here is what I got shortly after starting the dd test:
>
>
>
> [ 288.307993]
> [ 288.308004] =========================================================
> [ 288.308008] [ INFO: possible irq lock inversion dependency detected ]
> [ 288.308014] 3.18.0-rc6-ceph-00024-g72ca172 #1 Tainted: G E
> [ 288.308019] ---------------------------------------------------------
> [ 288.308023] kswapd1/87 just changed the state of lock:
> [ 288.308027] (&xfs_dir_ilock_class){++++-+}, at: [<ffffffffa0682d44>]
> xfs_ilock+0x134/0x160 [xfs]
> [ 288.308072] but this lock took another, RECLAIM_FS-unsafe lock in the
> past:
> [ 288.308076] (&mm->mmap_sem){++++++}
> [ 288.308076]
> [ 288.308076] and interrupts could create inverse lock ordering between
> them.
> [ 288.308076]
> [ 288.308084]
> [ 288.308084] other info that might help us debug this:
> [ 288.308089] Possible interrupt unsafe locking scenario:
> [ 288.308089]
> [ 288.308094] CPU0 CPU1
> [ 288.308097] ---- ----
> [ 288.308100] lock(&mm->mmap_sem);
> [ 288.308104] local_irq_disable();
> [ 288.308109] lock(&xfs_dir_ilock_class);
> [ 288.308114] lock(&mm->mmap_sem);
> [ 288.308120] <Interrupt>
> [ 288.308122] lock(&xfs_dir_ilock_class);
> [ 288.308127]
> [ 288.308127] *** DEADLOCK ***
> [ 288.308127]
> [ 288.308133] 3 locks held by kswapd1/87:
> [ 288.308136] #0: (shrinker_rwsem){++++..}, at: [<ffffffff8117551f>]
> shrink_slab+0x3f/0x140
> [ 288.308151] #1: (&type->s_umount_key#27){++++.+}, at:
> [<ffffffff811d8c14>] grab_super_passive+0x44/0x90
> [ 288.308165] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at:
> [<ffffffffa067acd4>] xfs_reclaim_inodes_ag+0xb4/0x400 [xfs]
> [ 288.308192]
> [ 288.308192] the shortest dependencies between 2nd lock and 1st lock:
> [ 288.308206] -> (&mm->mmap_sem){++++++} ops: 27039227 {
> [ 288.308214] HARDIRQ-ON-W at:
> [ 288.308218] [<ffffffff810a7209>]
> __lock_acquire+0x629/0x1c90
> [ 288.308229] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308236] [<ffffffff8173ae99>]
> down_write+0x49/0x80
> [ 288.308244] [<ffffffff811dcd03>]
> do_execve_common.isra.25+0x283/0x6e0
> [ 288.308253] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308259] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308269] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308276] HARDIRQ-ON-R at:
> [ 288.308280] [<ffffffff810a6f23>]
> __lock_acquire+0x343/0x1c90
> [ 288.308287] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308294] [<ffffffff8118d833>]
> might_fault+0x93/0xc0
> [ 288.308304] [<ffffffff813b7a80>]
> __clear_user+0x20/0x70
> [ 288.308314] [<ffffffff813b7afe>]
> clear_user+0x2e/0x40
> [ 288.308320] [<ffffffff8122a4cd>] padzero+0x2d/0x40
> [ 288.308329] [<ffffffff8122b0bf>]
> load_elf_binary+0x9cf/0x1880
> [ 288.308336] [<ffffffff811db9f0>]
> search_binary_handler+0xa0/0x1e0
> [ 288.308343] [<ffffffff811dcfa2>]
> do_execve_common.isra.25+0x522/0x6e0
> [ 288.308351] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308358] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308366] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308373] SOFTIRQ-ON-W at:
> [ 288.308376] [<ffffffff810a6f54>]
> __lock_acquire+0x374/0x1c90
> [ 288.308384] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308391] [<ffffffff8173ae99>]
> down_write+0x49/0x80
> [ 288.308398] [<ffffffff811dcd03>]
> do_execve_common.isra.25+0x283/0x6e0
> [ 288.308406] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308412] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308420] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308427] SOFTIRQ-ON-R at:
> [ 288.308431] [<ffffffff810a6f54>]
> __lock_acquire+0x374/0x1c90
> [ 288.308438] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308445] [<ffffffff8118d833>]
> might_fault+0x93/0xc0
> [ 288.308452] [<ffffffff813b7a80>]
> __clear_user+0x20/0x70
> [ 288.308458] [<ffffffff813b7afe>]
> clear_user+0x2e/0x40
> [ 288.308464] [<ffffffff8122a4cd>] padzero+0x2d/0x40
> [ 288.308470] [<ffffffff8122b0bf>]
> load_elf_binary+0x9cf/0x1880
> [ 288.308477] [<ffffffff811db9f0>]
> search_binary_handler+0xa0/0x1e0
> [ 288.308485] [<ffffffff811dcfa2>]
> do_execve_common.isra.25+0x522/0x6e0
> [ 288.308493] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308499] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308507] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308514] RECLAIM_FS-ON-W at:
> [ 288.308518] [<ffffffff810a96fa>]
> mark_held_locks+0x6a/0x90
> [ 288.308526] [<ffffffff810a9e17>]
> lockdep_trace_alloc+0x67/0xd0
> [ 288.308534] [<ffffffff8116b394>]
> __alloc_pages_nodemask+0x84/0xba0
> [ 288.308544] [<ffffffff811af917>]
> alloc_pages_current+0xf7/0x170
> [ 288.308553] [<ffffffff81190729>]
> __pmd_alloc+0x39/0x160
> [ 288.308559] [<ffffffff8119a69e>]
> move_page_tables+0x62e/0x6b0
> [ 288.308566] [<ffffffff811db1cb>]
> shift_arg_pages+0xcb/0x1a0
> [ 288.308574] [<ffffffff811db47b>]
> setup_arg_pages+0x1db/0x200
> [ 288.308581] [<ffffffff8122ab45>]
> load_elf_binary+0x455/0x1880
> [ 288.308588] [<ffffffff811db9f0>]
> search_binary_handler+0xa0/0x1e0
> [ 288.308596] [<ffffffff811dcfa2>]
> do_execve_common.isra.25+0x522/0x6e0
> [ 288.308603] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308610] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308617] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308624] RECLAIM_FS-ON-R at:
> [ 288.308628] [<ffffffff810a96fa>]
> mark_held_locks+0x6a/0x90
> [ 288.308635] [<ffffffff810a9e17>]
> lockdep_trace_alloc+0x67/0xd0
> [ 288.308643] [<ffffffff8116b394>]
> __alloc_pages_nodemask+0x84/0xba0
> [ 288.308651] [<ffffffff811af917>]
> alloc_pages_current+0xf7/0x170
> [ 288.308658] [<ffffffff81166bee>]
> __get_free_pages+0xe/0x40
> [ 288.308666] [<ffffffff81166c36>]
> get_zeroed_page+0x16/0x20
> [ 288.308673] [<ffffffff8118ff89>]
> __pud_alloc+0x29/0xb0
> [ 288.308679] [<ffffffff81191873>]
> handle_mm_fault+0xec3/0x10b0
> [ 288.308686] [<ffffffff81047f32>]
> __do_page_fault+0x1c2/0x580
> [ 288.308694] [<ffffffff8104844c>]
> do_page_fault+0xc/0x10
> [ 288.308700] [<ffffffff8173f4b2>]
> page_fault+0x22/0x30
> [ 288.308707] [<ffffffff813b7afe>]
> clear_user+0x2e/0x40
> [ 288.308713] [<ffffffff8122a4cd>]
> padzero+0x2d/0x40
> [ 288.308719] [<ffffffff8122b0bf>]
> load_elf_binary+0x9cf/0x1880
> [ 288.308726] [<ffffffff811db9f0>]
> search_binary_handler+0xa0/0x1e0
> [ 288.308734] [<ffffffff811dcfa2>]
> do_execve_common.isra.25+0x522/0x6e0
> [ 288.308741] [<ffffffff811dd178>]
> do_execve+0x18/0x20
> [ 288.308748] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308755] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308762] INITIAL USE at:
> [ 288.308766] [<ffffffff810a6ff1>]
> __lock_acquire+0x411/0x1c90
> [ 288.308773] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308779] [<ffffffff8173ae99>]
> down_write+0x49/0x80
> [ 288.308785] [<ffffffff811dcd03>]
> do_execve_common.isra.25+0x283/0x6e0
> [ 288.308793] [<ffffffff811dd178>] do_execve+0x18/0x20
> [ 288.308800] [<ffffffff8106ff4e>]
> ____call_usermodehelper+0x11e/0x170
> [ 288.308807] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.308814] }
> [ 288.308816] ... key at: [<ffffffff81eaa388>] __key.49413+0x0/0x8
> [ 288.308824] ... acquired at:
> [ 288.308827] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> [ 288.308833] [<ffffffff8118d833>] might_fault+0x93/0xc0
> [ 288.308839] [<ffffffff811e92b4>] filldir+0xa4/0x120
> [ 288.308845] [<ffffffffa0674308>] xfs_dir2_leaf_getdents+0x2d8/0x370
> [xfs]
> [ 288.308866] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
> [ 288.308884] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
> [ 288.308902] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
> [ 288.308908] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
> [ 288.308914] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
> [ 288.308920]
> [ 288.308922] -> (&xfs_dir_ilock_class){++++-+} ops: 233164 {
> [ 288.308930] HARDIRQ-ON-W at:
> [ 288.308933] [<ffffffff810a7209>]
> __lock_acquire+0x629/0x1c90
> [ 288.308941] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.308947] [<ffffffff810a33ef>]
> down_write_nested+0x4f/0x80
> [ 288.308955] [<ffffffffa0682d44>]
> xfs_ilock+0x134/0x160 [xfs]
> [ 288.308976] [<ffffffffa0685fcd>]
> xfs_inactive_ifree+0x8d/0x230 [xfs]
> [ 288.308997] [<ffffffffa0686229>]
> xfs_inactive+0xb9/0x170 [xfs]
> [ 288.309017] [<ffffffffa068c0d8>]
> xfs_fs_evict_inode+0xa8/0x110 [xfs]
> [ 288.309038] [<ffffffff811f1ed8>] evict+0xb8/0x190
> [ 288.309045] [<ffffffff811f2815>] iput+0x105/0x190
> [ 288.309050] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
> [ 288.309057] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
> [ 288.309063] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
> [ 288.309069] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
> [ 288.309075] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309083] HARDIRQ-ON-R at:
> [ 288.309086] [<ffffffff810a6f23>]
> __lock_acquire+0x343/0x1c90
> [ 288.309093] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.309100] [<ffffffff810a3387>]
> down_read_nested+0x57/0x70
> [ 288.309107] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> [xfs]
> [ 288.309126] [<ffffffffa0682df5>]
> xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> [ 288.309145] [<ffffffffa0683fe2>]
> xfs_lookup+0x62/0x160 [xfs]
> [ 288.309164] [<ffffffffa068095b>]
> xfs_vn_lookup+0x5b/0xa0 [xfs]
> [ 288.309183] [<ffffffff811df12d>]
> lookup_real+0x1d/0x60
> [ 288.309189] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> [ 288.309195] [<ffffffff811e4c44>]
> path_openat+0xc4/0x660
> [ 288.309201] [<ffffffff811e6a33>]
> do_filp_open+0x43/0xa0
> [ 288.309207] [<ffffffff811d4462>]
> do_sys_open+0x132/0x220
> [ 288.309213] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> [ 288.309219] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309226] SOFTIRQ-ON-W at:
> [ 288.309230] [<ffffffff810a6f54>]
> __lock_acquire+0x374/0x1c90
> [ 288.309237] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.309243] [<ffffffff810a33ef>]
> down_write_nested+0x4f/0x80
> [ 288.309250] [<ffffffffa0682d44>]
> xfs_ilock+0x134/0x160 [xfs]
> [ 288.309268] [<ffffffffa0685fcd>]
> xfs_inactive_ifree+0x8d/0x230 [xfs]
> [ 288.309287] [<ffffffffa0686229>]
> xfs_inactive+0xb9/0x170 [xfs]
> [ 288.309305] [<ffffffffa068c0d8>]
> xfs_fs_evict_inode+0xa8/0x110 [xfs]
> [ 288.309326] [<ffffffff811f1ed8>] evict+0xb8/0x190
> [ 288.309332] [<ffffffff811f2815>] iput+0x105/0x190
> [ 288.309338] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
> [ 288.309344] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
> [ 288.309350] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
> [ 288.309356] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
> [ 288.309362] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309369] SOFTIRQ-ON-R at:
> [ 288.309373] [<ffffffff810a6f54>]
> __lock_acquire+0x374/0x1c90
> [ 288.309380] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.309387] [<ffffffff810a3387>]
> down_read_nested+0x57/0x70
> [ 288.309393] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> [xfs]
> [ 288.309412] [<ffffffffa0682df5>]
> xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> [ 288.309431] [<ffffffffa0683fe2>]
> xfs_lookup+0x62/0x160 [xfs]
> [ 288.309449] [<ffffffffa068095b>]
> xfs_vn_lookup+0x5b/0xa0 [xfs]
> [ 288.309467] [<ffffffff811df12d>]
> lookup_real+0x1d/0x60
> [ 288.309473] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> [ 288.309479] [<ffffffff811e4c44>]
> path_openat+0xc4/0x660
> [ 288.309485] [<ffffffff811e6a33>]
> do_filp_open+0x43/0xa0
> [ 288.309491] [<ffffffff811d4462>]
> do_sys_open+0x132/0x220
> [ 288.309497] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> [ 288.309502] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309510] IN-RECLAIM_FS-W at:
> [ 288.309513] [<ffffffff810a6fb0>]
> __lock_acquire+0x3d0/0x1c90
> [ 288.309521] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.309528] [<ffffffff810a33ef>]
> down_write_nested+0x4f/0x80
> [ 288.309535] [<ffffffffa0682d44>]
> xfs_ilock+0x134/0x160 [xfs]
> [ 288.309552] [<ffffffffa067aa0c>]
> xfs_reclaim_inode+0x12c/0x340 [xfs]
> [ 288.309572] [<ffffffffa067aea7>]
> xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
> [ 288.309592] [<ffffffffa067bda3>]
> xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> [ 288.309611] [<ffffffffa068b855>]
> xfs_fs_free_cached_objects+0x15/0x20 [xfs]
> [ 288.309632] [<ffffffff811d8dd8>]
> super_cache_scan+0x178/0x180
> [ 288.309640] [<ffffffff8117393e>]
> shrink_slab_node+0x15e/0x310
> [ 288.309647] [<ffffffff811755e0>]
> shrink_slab+0x100/0x140
> [ 288.309654] [<ffffffff81178306>]
> kswapd_shrink_zone+0x116/0x1a0
> [ 288.309661] [<ffffffff8117925b>]
> kswapd+0x4bb/0x9a0
> [ 288.309667] [<ffffffff8107a664>]
> kthread+0xe4/0x100
> [ 288.309674] [<ffffffff8173d66c>]
> ret_from_fork+0x7c/0xb0
> [ 288.309682] RECLAIM_FS-ON-R at:
> [ 288.309685] [<ffffffff810a96fa>]
> mark_held_locks+0x6a/0x90
> [ 288.309693] [<ffffffff810a9e17>]
> lockdep_trace_alloc+0x67/0xd0
> [ 288.309700] [<ffffffff8116b394>]
> __alloc_pages_nodemask+0x84/0xba0
> [ 288.309708] [<ffffffff811b1503>]
> alloc_pages_vma+0xf3/0x1c0
> [ 288.309715] [<ffffffff811916ce>]
> handle_mm_fault+0xd1e/0x10b0
> [ 288.309722] [<ffffffff81047f32>]
> __do_page_fault+0x1c2/0x580
> [ 288.309729] [<ffffffff8104844c>]
> do_page_fault+0xc/0x10
> [ 288.309735] [<ffffffff8173f4b2>]
> page_fault+0x22/0x30
> [ 288.309741] [<ffffffffa0674308>]
> xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
> [ 288.309761] [<ffffffffa0674556>]
> xfs_readdir+0x1b6/0x1c0 [xfs]
> [ 288.309780] [<ffffffffa067606b>]
> xfs_file_readdir+0x2b/0x30 [xfs]
> [ 288.309799] [<ffffffff811e93ee>]
> iterate_dir+0xbe/0x150
> [ 288.309806] [<ffffffff811e959c>]
> SyS_getdents+0x8c/0x110
> [ 288.309813] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309821] INITIAL USE at:
> [ 288.309824] [<ffffffff810a6ff1>]
> __lock_acquire+0x411/0x1c90
> [ 288.309831] [<ffffffff810a8e9e>]
> lock_acquire+0x9e/0x140
> [ 288.309837] [<ffffffff810a3387>]
> down_read_nested+0x57/0x70
> [ 288.309844] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> [xfs]
> [ 288.309863] [<ffffffffa0682df5>]
> xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> [ 288.309882] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160
> [xfs]
> [ 288.309900] [<ffffffffa068095b>]
> xfs_vn_lookup+0x5b/0xa0 [xfs]
> [ 288.309919] [<ffffffff811df12d>] lookup_real+0x1d/0x60
> [ 288.309924] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> [ 288.309930] [<ffffffff811e4c44>]
> path_openat+0xc4/0x660
> [ 288.309936] [<ffffffff811e6a33>]
> do_filp_open+0x43/0xa0
> [ 288.309941] [<ffffffff811d4462>]
> do_sys_open+0x132/0x220
> [ 288.309947] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> [ 288.309953] [<ffffffff8173d716>]
> system_call_fastpath+0x16/0x1b
> [ 288.309960] }
> [ 288.309962] ... key at: [<ffffffffa06decf4>]
> xfs_dir_ilock_class+0x0/0xfffffffffffd130c [xfs]
> [ 288.309983] ... acquired at:
> [ 288.309985] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
> [ 288.309992] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
> [ 288.309997] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
> [ 288.310003] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> [ 288.310009] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
> [ 288.310015] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
> [ 288.310032] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
> [ 288.310050] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400
> [xfs]
> [ 288.310068] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> [ 288.310086] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20
> [xfs]
> [ 288.310107] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
> [ 288.310113] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
> [ 288.310118] [<ffffffff811755e0>] shrink_slab+0x100/0x140
> [ 288.310124] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
> [ 288.310130] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
> [ 288.310135] [<ffffffff8107a664>] kthread+0xe4/0x100
> [ 288.310141] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
> [ 288.310147]
> [ 288.310149]
> [ 288.310149] stack backtrace:
> [ 288.310156] CPU: 8 PID: 87 Comm: kswapd1 Tainted: G E
> 3.18.0-rc6-ceph-00024-g72ca172 #1
> [ 288.310162] Hardware name: Supermicro
> X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
> [ 288.310169] ffffffff821208e0 ffff8804676ab608 ffffffff81733b38
> 0000000000000007
> [ 288.310177] ffff8804676ab670 ffff8804676ab658 ffffffff810a5f68
> ffffffff821208e0
> [ 288.310184] ffffffff81a7cbe0 ffff8804676ab674 0000000000000000
> ffff88046763cc50
> [ 288.310192] Call Trace:
> [ 288.310200] [<ffffffff81733b38>] dump_stack+0x4e/0x68
> [ 288.310206] [<ffffffff810a5f68>]
> print_irq_inversion_bug.part.41+0x1e8/0x1f0
> [ 288.310213] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
> [ 288.310220] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
> [ 288.310226] [<ffffffff810a5f70>] ?
> print_irq_inversion_bug.part.41+0x1f0/0x1f0
> [ 288.310234] [<ffffffff811c9185>] ? __mem_cgroup_threshold+0x5/0x1d0
> [ 288.310241] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
> [ 288.310247] [<ffffffff810a6ff1>] ? __lock_acquire+0x411/0x1c90
> [ 288.310266] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> [ 288.310272] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> [ 288.310289] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> [ 288.310295] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
> [ 288.310312] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> [ 288.310329] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
> [ 288.310347] [<ffffffffa067aa0c>] ? xfs_reclaim_inode+0x12c/0x340 [xfs]
> [ 288.310364] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
> [ 288.310382] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
> [ 288.310400] [<ffffffffa067ad00>] ? xfs_reclaim_inodes_ag+0xe0/0x400
> [xfs]
> [ 288.310418] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> [ 288.310438] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20
> [xfs]
> [ 288.310445] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
> [ 288.310451] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
> [ 288.310457] [<ffffffff811755e0>] shrink_slab+0x100/0x140
> [ 288.310463] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
> [ 288.310469] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
> [ 288.310475] [<ffffffff81178da0>] ?
> mem_cgroup_shrink_node_zone+0x1c0/0x1c0
> [ 288.310481] [<ffffffff8107a664>] kthread+0xe4/0x100
> [ 288.310488] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
> [ 288.310494] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
> [ 288.310500] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
>
>
>
> I've not seen any hang tasks just yet. The server seems to continue working.
> I will do more testing and get back to you with more info.
Yeah, it's not related to rbd.
Thanks,
Ilya
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
