Ilya, here is what I got shortly after starting the dd test:
[ 288.307993]
[ 288.308004] =========================================================
[ 288.308008] [ INFO: possible irq lock inversion dependency detected ]
[ 288.308014] 3.18.0-rc6-ceph-00024-g72ca172 #1 Tainted: G E
[ 288.308019] ---------------------------------------------------------
[ 288.308023] kswapd1/87 just changed the state of lock:
[ 288.308027] (&xfs_dir_ilock_class){++++-+}, at: [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.308072] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[ 288.308076] (&mm->mmap_sem){++++++}
[ 288.308076]
[ 288.308076] and interrupts could create inverse lock ordering between them.
[ 288.308076]
[ 288.308084]
[ 288.308084] other info that might help us debug this:
[ 288.308089] Possible interrupt unsafe locking scenario:
[ 288.308089]
[ 288.308094] CPU0 CPU1
[ 288.308097] ---- ----
[ 288.308100] lock(&mm->mmap_sem);
[ 288.308104] local_irq_disable();
[ 288.308109] lock(&xfs_dir_ilock_class);
[ 288.308114] lock(&mm->mmap_sem);
[ 288.308120] <Interrupt>
[ 288.308122] lock(&xfs_dir_ilock_class);
[ 288.308127]
[ 288.308127] *** DEADLOCK ***
[ 288.308127]
[ 288.308133] 3 locks held by kswapd1/87:
[ 288.308136] #0: (shrinker_rwsem){++++..}, at: [<ffffffff8117551f>] shrink_slab+0x3f/0x140
[ 288.308151] #1: (&type->s_umount_key#27){++++.+}, at: [<ffffffff811d8c14>] grab_super_passive+0x44/0x90
[ 288.308165] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffffa067acd4>] xfs_reclaim_inodes_ag+0xb4/0x400 [xfs]
[ 288.308192]
[ 288.308192] the shortest dependencies between 2nd lock and 1st lock:
[ 288.308206] -> (&mm->mmap_sem){++++++} ops: 27039227 {
[ 288.308214] HARDIRQ-ON-W at:
[ 288.308218] [<ffffffff810a7209>] __lock_acquire+0x629/0x1c90
[ 288.308229] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308236] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308244] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308253] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308259] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308269] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308276] HARDIRQ-ON-R at:
[ 288.308280] [<ffffffff810a6f23>] __lock_acquire+0x343/0x1c90
[ 288.308287] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308294] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308304] [<ffffffff813b7a80>] __clear_user+0x20/0x70
[ 288.308314] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308320] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308329] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308336] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308343] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308351] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308358] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308366] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308373] SOFTIRQ-ON-W at:
[ 288.308376] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.308384] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308391] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308398] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308406] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308412] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308420] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308427] SOFTIRQ-ON-R at:
[ 288.308431] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.308438] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308445] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308452] [<ffffffff813b7a80>] __clear_user+0x20/0x70
[ 288.308458] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308464] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308470] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308477] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308485] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308493] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308499] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308507] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308514] RECLAIM_FS-ON-W at:
[ 288.308518] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.308526] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.308534] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.308544] [<ffffffff811af917>] alloc_pages_current+0xf7/0x170
[ 288.308553] [<ffffffff81190729>] __pmd_alloc+0x39/0x160
[ 288.308559] [<ffffffff8119a69e>] move_page_tables+0x62e/0x6b0
[ 288.308566] [<ffffffff811db1cb>] shift_arg_pages+0xcb/0x1a0
[ 288.308574] [<ffffffff811db47b>] setup_arg_pages+0x1db/0x200
[ 288.308581] [<ffffffff8122ab45>] load_elf_binary+0x455/0x1880
[ 288.308588] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308596] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308603] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308610] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308617] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308624] RECLAIM_FS-ON-R at:
[ 288.308628] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.308635] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.308643] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.308651] [<ffffffff811af917>] alloc_pages_current+0xf7/0x170
[ 288.308658] [<ffffffff81166bee>] __get_free_pages+0xe/0x40
[ 288.308666] [<ffffffff81166c36>] get_zeroed_page+0x16/0x20
[ 288.308673] [<ffffffff8118ff89>] __pud_alloc+0x29/0xb0
[ 288.308679] [<ffffffff81191873>] handle_mm_fault+0xec3/0x10b0
[ 288.308686] [<ffffffff81047f32>] __do_page_fault+0x1c2/0x580
[ 288.308694] [<ffffffff8104844c>] do_page_fault+0xc/0x10
[ 288.308700] [<ffffffff8173f4b2>] page_fault+0x22/0x30
[ 288.308707] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308713] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308719] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308726] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308734] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308741] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308748] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308755] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308762] INITIAL USE at:
[ 288.308766] [<ffffffff810a6ff1>] __lock_acquire+0x411/0x1c90
[ 288.308773] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308779] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308785] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308793] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308800] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308807] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308814] }
[ 288.308816] ... key at: [<ffffffff81eaa388>] __key.49413+0x0/0x8
[ 288.308824] ... acquired at:
[ 288.308827] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308833] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308839] [<ffffffff811e92b4>] filldir+0xa4/0x120
[ 288.308845] [<ffffffffa0674308>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
[ 288.308866] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
[ 288.308884] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
[ 288.308902] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
[ 288.308908] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
[ 288.308914] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.308920]
[ 288.308922] -> (&xfs_dir_ilock_class){++++-+} ops: 233164 {
[ 288.308930] HARDIRQ-ON-W at:
[ 288.308933] [<ffffffff810a7209>] __lock_acquire+0x629/0x1c90
[ 288.308941] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308947] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.308955] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.308976] [<ffffffffa0685fcd>] xfs_inactive_ifree+0x8d/0x230 [xfs]
[ 288.308997] [<ffffffffa0686229>] xfs_inactive+0xb9/0x170 [xfs]
[ 288.309017] [<ffffffffa068c0d8>] xfs_fs_evict_inode+0xa8/0x110 [xfs]
[ 288.309038] [<ffffffff811f1ed8>] evict+0xb8/0x190
[ 288.309045] [<ffffffff811f2815>] iput+0x105/0x190
[ 288.309050] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
[ 288.309057] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
[ 288.309063] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
[ 288.309069] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
[ 288.309075] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309083] HARDIRQ-ON-R at:
[ 288.309086] [<ffffffff810a6f23>] __lock_acquire+0x343/0x1c90
[ 288.309093] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309100] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309107] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309126] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309145] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309164] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309183] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309189] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309195] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309201] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309207] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309213] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309219] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309226] SOFTIRQ-ON-W at:
[ 288.309230] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.309237] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309243] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.309250] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.309268] [<ffffffffa0685fcd>] xfs_inactive_ifree+0x8d/0x230 [xfs]
[ 288.309287] [<ffffffffa0686229>] xfs_inactive+0xb9/0x170 [xfs]
[ 288.309305] [<ffffffffa068c0d8>] xfs_fs_evict_inode+0xa8/0x110 [xfs]
[ 288.309326] [<ffffffff811f1ed8>] evict+0xb8/0x190
[ 288.309332] [<ffffffff811f2815>] iput+0x105/0x190
[ 288.309338] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
[ 288.309344] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
[ 288.309350] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
[ 288.309356] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
[ 288.309362] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309369] SOFTIRQ-ON-R at:
[ 288.309373] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.309380] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309387] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309393] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309412] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309431] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309449] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309467] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309473] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309479] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309485] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309491] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309497] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309502] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309510] IN-RECLAIM_FS-W at:
[ 288.309513] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.309521] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309528] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.309535] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.309552] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.309572] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.309592] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.309611] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.309632] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.309640] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.309647] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.309654] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.309661] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.309667] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.309674] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.309682] RECLAIM_FS-ON-R at:
[ 288.309685] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.309693] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.309700] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.309708] [<ffffffff811b1503>] alloc_pages_vma+0xf3/0x1c0
[ 288.309715] [<ffffffff811916ce>] handle_mm_fault+0xd1e/0x10b0
[ 288.309722] [<ffffffff81047f32>] __do_page_fault+0x1c2/0x580
[ 288.309729] [<ffffffff8104844c>] do_page_fault+0xc/0x10
[ 288.309735] [<ffffffff8173f4b2>] page_fault+0x22/0x30
[ 288.309741] [<ffffffffa0674308>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
[ 288.309761] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
[ 288.309780] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
[ 288.309799] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
[ 288.309806] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
[ 288.309813] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309821] INITIAL USE at:
[ 288.309824] [<ffffffff810a6ff1>] __lock_acquire+0x411/0x1c90
[ 288.309831] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309837] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309844] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309863] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309882] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309900] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309919] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309924] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309930] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309936] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309941] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309947] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309953] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309960] }
[ 288.309962] ... key at: [<ffffffffa06decf4>] xfs_dir_ilock_class+0x0/0xfffffffffffd130c [xfs]
[ 288.309983] ... acquired at:
[ 288.309985] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
[ 288.309992] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
[ 288.309997] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.310003] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.310009] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.310015] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.310032] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310050] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.310068] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.310086] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.310107] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.310113] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.310118] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.310124] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.310130] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.310135] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.310141] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.310147]
[ 288.310149]
[ 288.310149] stack backtrace:
[ 288.310156] CPU: 8 PID: 87 Comm: kswapd1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1
[ 288.310162] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[ 288.310169] ffffffff821208e0 ffff8804676ab608 ffffffff81733b38 0000000000000007
[ 288.310177] ffff8804676ab670 ffff8804676ab658 ffffffff810a5f68 ffffffff821208e0
[ 288.310184] ffffffff81a7cbe0 ffff8804676ab674 0000000000000000 ffff88046763cc50
[ 288.310192] Call Trace:
[ 288.310200] [<ffffffff81733b38>] dump_stack+0x4e/0x68
[ 288.310206] [<ffffffff810a5f68>] print_irq_inversion_bug.part.41+0x1e8/0x1f0
[ 288.310213] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
[ 288.310220] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
[ 288.310226] [<ffffffff810a5f70>] ? print_irq_inversion_bug.part.41+0x1f0/0x1f0
[ 288.310234] [<ffffffff811c9185>] ? __mem_cgroup_threshold+0x5/0x1d0
[ 288.310241] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.310247] [<ffffffff810a6ff1>] ? __lock_acquire+0x411/0x1c90
[ 288.310266] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310272] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.310289] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310295] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.310312] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310329] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.310347] [<ffffffffa067aa0c>] ? xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310364] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310382] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.310400] [<ffffffffa067ad00>] ? xfs_reclaim_inodes_ag+0xe0/0x400 [xfs]
[ 288.310418] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.310438] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.310445] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.310451] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.310457] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.310463] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.310469] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.310475] [<ffffffff81178da0>] ? mem_cgroup_shrink_node_zone+0x1c0/0x1c0
[ 288.310481] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.310488] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
[ 288.310494] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.310500] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
I've not seen any hang tasks just yet. The server seems to continue working. I will do more testing and get back to you with more info.
Andrei
[ 288.307993]
[ 288.308004] =========================================================
[ 288.308008] [ INFO: possible irq lock inversion dependency detected ]
[ 288.308014] 3.18.0-rc6-ceph-00024-g72ca172 #1 Tainted: G E
[ 288.308019] ---------------------------------------------------------
[ 288.308023] kswapd1/87 just changed the state of lock:
[ 288.308027] (&xfs_dir_ilock_class){++++-+}, at: [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.308072] but this lock took another, RECLAIM_FS-unsafe lock in the past:
[ 288.308076] (&mm->mmap_sem){++++++}
[ 288.308076]
[ 288.308076] and interrupts could create inverse lock ordering between them.
[ 288.308076]
[ 288.308084]
[ 288.308084] other info that might help us debug this:
[ 288.308089] Possible interrupt unsafe locking scenario:
[ 288.308089]
[ 288.308094] CPU0 CPU1
[ 288.308097] ---- ----
[ 288.308100] lock(&mm->mmap_sem);
[ 288.308104] local_irq_disable();
[ 288.308109] lock(&xfs_dir_ilock_class);
[ 288.308114] lock(&mm->mmap_sem);
[ 288.308120] <Interrupt>
[ 288.308122] lock(&xfs_dir_ilock_class);
[ 288.308127]
[ 288.308127] *** DEADLOCK ***
[ 288.308127]
[ 288.308133] 3 locks held by kswapd1/87:
[ 288.308136] #0: (shrinker_rwsem){++++..}, at: [<ffffffff8117551f>] shrink_slab+0x3f/0x140
[ 288.308151] #1: (&type->s_umount_key#27){++++.+}, at: [<ffffffff811d8c14>] grab_super_passive+0x44/0x90
[ 288.308165] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffffa067acd4>] xfs_reclaim_inodes_ag+0xb4/0x400 [xfs]
[ 288.308192]
[ 288.308192] the shortest dependencies between 2nd lock and 1st lock:
[ 288.308206] -> (&mm->mmap_sem){++++++} ops: 27039227 {
[ 288.308214] HARDIRQ-ON-W at:
[ 288.308218] [<ffffffff810a7209>] __lock_acquire+0x629/0x1c90
[ 288.308229] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308236] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308244] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308253] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308259] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308269] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308276] HARDIRQ-ON-R at:
[ 288.308280] [<ffffffff810a6f23>] __lock_acquire+0x343/0x1c90
[ 288.308287] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308294] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308304] [<ffffffff813b7a80>] __clear_user+0x20/0x70
[ 288.308314] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308320] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308329] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308336] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308343] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308351] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308358] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308366] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308373] SOFTIRQ-ON-W at:
[ 288.308376] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.308384] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308391] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308398] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308406] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308412] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308420] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308427] SOFTIRQ-ON-R at:
[ 288.308431] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.308438] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308445] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308452] [<ffffffff813b7a80>] __clear_user+0x20/0x70
[ 288.308458] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308464] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308470] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308477] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308485] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308493] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308499] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308507] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308514] RECLAIM_FS-ON-W at:
[ 288.308518] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.308526] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.308534] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.308544] [<ffffffff811af917>] alloc_pages_current+0xf7/0x170
[ 288.308553] [<ffffffff81190729>] __pmd_alloc+0x39/0x160
[ 288.308559] [<ffffffff8119a69e>] move_page_tables+0x62e/0x6b0
[ 288.308566] [<ffffffff811db1cb>] shift_arg_pages+0xcb/0x1a0
[ 288.308574] [<ffffffff811db47b>] setup_arg_pages+0x1db/0x200
[ 288.308581] [<ffffffff8122ab45>] load_elf_binary+0x455/0x1880
[ 288.308588] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308596] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308603] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308610] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308617] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308624] RECLAIM_FS-ON-R at:
[ 288.308628] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.308635] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.308643] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.308651] [<ffffffff811af917>] alloc_pages_current+0xf7/0x170
[ 288.308658] [<ffffffff81166bee>] __get_free_pages+0xe/0x40
[ 288.308666] [<ffffffff81166c36>] get_zeroed_page+0x16/0x20
[ 288.308673] [<ffffffff8118ff89>] __pud_alloc+0x29/0xb0
[ 288.308679] [<ffffffff81191873>] handle_mm_fault+0xec3/0x10b0
[ 288.308686] [<ffffffff81047f32>] __do_page_fault+0x1c2/0x580
[ 288.308694] [<ffffffff8104844c>] do_page_fault+0xc/0x10
[ 288.308700] [<ffffffff8173f4b2>] page_fault+0x22/0x30
[ 288.308707] [<ffffffff813b7afe>] clear_user+0x2e/0x40
[ 288.308713] [<ffffffff8122a4cd>] padzero+0x2d/0x40
[ 288.308719] [<ffffffff8122b0bf>] load_elf_binary+0x9cf/0x1880
[ 288.308726] [<ffffffff811db9f0>] search_binary_handler+0xa0/0x1e0
[ 288.308734] [<ffffffff811dcfa2>] do_execve_common.isra.25+0x522/0x6e0
[ 288.308741] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308748] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308755] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308762] INITIAL USE at:
[ 288.308766] [<ffffffff810a6ff1>] __lock_acquire+0x411/0x1c90
[ 288.308773] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308779] [<ffffffff8173ae99>] down_write+0x49/0x80
[ 288.308785] [<ffffffff811dcd03>] do_execve_common.isra.25+0x283/0x6e0
[ 288.308793] [<ffffffff811dd178>] do_execve+0x18/0x20
[ 288.308800] [<ffffffff8106ff4e>] ____call_usermodehelper+0x11e/0x170
[ 288.308807] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.308814] }
[ 288.308816] ... key at: [<ffffffff81eaa388>] __key.49413+0x0/0x8
[ 288.308824] ... acquired at:
[ 288.308827] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308833] [<ffffffff8118d833>] might_fault+0x93/0xc0
[ 288.308839] [<ffffffff811e92b4>] filldir+0xa4/0x120
[ 288.308845] [<ffffffffa0674308>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
[ 288.308866] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
[ 288.308884] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
[ 288.308902] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
[ 288.308908] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
[ 288.308914] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.308920]
[ 288.308922] -> (&xfs_dir_ilock_class){++++-+} ops: 233164 {
[ 288.308930] HARDIRQ-ON-W at:
[ 288.308933] [<ffffffff810a7209>] __lock_acquire+0x629/0x1c90
[ 288.308941] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.308947] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.308955] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.308976] [<ffffffffa0685fcd>] xfs_inactive_ifree+0x8d/0x230 [xfs]
[ 288.308997] [<ffffffffa0686229>] xfs_inactive+0xb9/0x170 [xfs]
[ 288.309017] [<ffffffffa068c0d8>] xfs_fs_evict_inode+0xa8/0x110 [xfs]
[ 288.309038] [<ffffffff811f1ed8>] evict+0xb8/0x190
[ 288.309045] [<ffffffff811f2815>] iput+0x105/0x190
[ 288.309050] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
[ 288.309057] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
[ 288.309063] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
[ 288.309069] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
[ 288.309075] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309083] HARDIRQ-ON-R at:
[ 288.309086] [<ffffffff810a6f23>] __lock_acquire+0x343/0x1c90
[ 288.309093] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309100] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309107] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309126] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309145] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309164] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309183] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309189] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309195] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309201] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309207] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309213] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309219] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309226] SOFTIRQ-ON-W at:
[ 288.309230] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.309237] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309243] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.309250] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.309268] [<ffffffffa0685fcd>] xfs_inactive_ifree+0x8d/0x230 [xfs]
[ 288.309287] [<ffffffffa0686229>] xfs_inactive+0xb9/0x170 [xfs]
[ 288.309305] [<ffffffffa068c0d8>] xfs_fs_evict_inode+0xa8/0x110 [xfs]
[ 288.309326] [<ffffffff811f1ed8>] evict+0xb8/0x190
[ 288.309332] [<ffffffff811f2815>] iput+0x105/0x190
[ 288.309338] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
[ 288.309344] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
[ 288.309350] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
[ 288.309356] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
[ 288.309362] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309369] SOFTIRQ-ON-R at:
[ 288.309373] [<ffffffff810a6f54>] __lock_acquire+0x374/0x1c90
[ 288.309380] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309387] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309393] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309412] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309431] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309449] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309467] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309473] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309479] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309485] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309491] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309497] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309502] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309510] IN-RECLAIM_FS-W at:
[ 288.309513] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.309521] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309528] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.309535] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.309552] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.309572] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.309592] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.309611] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.309632] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.309640] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.309647] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.309654] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.309661] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.309667] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.309674] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.309682] RECLAIM_FS-ON-R at:
[ 288.309685] [<ffffffff810a96fa>] mark_held_locks+0x6a/0x90
[ 288.309693] [<ffffffff810a9e17>] lockdep_trace_alloc+0x67/0xd0
[ 288.309700] [<ffffffff8116b394>] __alloc_pages_nodemask+0x84/0xba0
[ 288.309708] [<ffffffff811b1503>] alloc_pages_vma+0xf3/0x1c0
[ 288.309715] [<ffffffff811916ce>] handle_mm_fault+0xd1e/0x10b0
[ 288.309722] [<ffffffff81047f32>] __do_page_fault+0x1c2/0x580
[ 288.309729] [<ffffffff8104844c>] do_page_fault+0xc/0x10
[ 288.309735] [<ffffffff8173f4b2>] page_fault+0x22/0x30
[ 288.309741] [<ffffffffa0674308>] xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
[ 288.309761] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
[ 288.309780] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
[ 288.309799] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
[ 288.309806] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
[ 288.309813] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309821] INITIAL USE at:
[ 288.309824] [<ffffffff810a6ff1>] __lock_acquire+0x411/0x1c90
[ 288.309831] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.309837] [<ffffffff810a3387>] down_read_nested+0x57/0x70
[ 288.309844] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160 [xfs]
[ 288.309863] [<ffffffffa0682df5>] xfs_ilock_data_map_shared+0x35/0x40 [xfs]
[ 288.309882] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160 [xfs]
[ 288.309900] [<ffffffffa068095b>] xfs_vn_lookup+0x5b/0xa0 [xfs]
[ 288.309919] [<ffffffff811df12d>] lookup_real+0x1d/0x60
[ 288.309924] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
[ 288.309930] [<ffffffff811e4c44>] path_openat+0xc4/0x660
[ 288.309936] [<ffffffff811e6a33>] do_filp_open+0x43/0xa0
[ 288.309941] [<ffffffff811d4462>] do_sys_open+0x132/0x220
[ 288.309947] [<ffffffff811d456e>] SyS_open+0x1e/0x20
[ 288.309953] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
[ 288.309960] }
[ 288.309962] ... key at: [<ffffffffa06decf4>] xfs_dir_ilock_class+0x0/0xfffffffffffd130c [xfs]
[ 288.309983] ... acquired at:
[ 288.309985] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
[ 288.309992] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
[ 288.309997] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.310003] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.310009] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.310015] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.310032] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310050] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.310068] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.310086] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.310107] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.310113] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.310118] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.310124] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.310130] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.310135] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.310141] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.310147]
[ 288.310149]
[ 288.310149] stack backtrace:
[ 288.310156] CPU: 8 PID: 87 Comm: kswapd1 Tainted: G E 3.18.0-rc6-ceph-00024-g72ca172 #1
[ 288.310162] Hardware name: Supermicro X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
[ 288.310169] ffffffff821208e0 ffff8804676ab608 ffffffff81733b38 0000000000000007
[ 288.310177] ffff8804676ab670 ffff8804676ab658 ffffffff810a5f68 ffffffff821208e0
[ 288.310184] ffffffff81a7cbe0 ffff8804676ab674 0000000000000000 ffff88046763cc50
[ 288.310192] Call Trace:
[ 288.310200] [<ffffffff81733b38>] dump_stack+0x4e/0x68
[ 288.310206] [<ffffffff810a5f68>] print_irq_inversion_bug.part.41+0x1e8/0x1f0
[ 288.310213] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
[ 288.310220] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
[ 288.310226] [<ffffffff810a5f70>] ? print_irq_inversion_bug.part.41+0x1f0/0x1f0
[ 288.310234] [<ffffffff811c9185>] ? __mem_cgroup_threshold+0x5/0x1d0
[ 288.310241] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
[ 288.310247] [<ffffffff810a6ff1>] ? __lock_acquire+0x411/0x1c90
[ 288.310266] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310272] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
[ 288.310289] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310295] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
[ 288.310312] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
[ 288.310329] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
[ 288.310347] [<ffffffffa067aa0c>] ? xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310364] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340 [xfs]
[ 288.310382] [<ffffffffa067aea7>] xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
[ 288.310400] [<ffffffffa067ad00>] ? xfs_reclaim_inodes_ag+0xe0/0x400 [xfs]
[ 288.310418] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[ 288.310438] [<ffffffffa068b855>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[ 288.310445] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
[ 288.310451] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
[ 288.310457] [<ffffffff811755e0>] shrink_slab+0x100/0x140
[ 288.310463] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
[ 288.310469] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
[ 288.310475] [<ffffffff81178da0>] ? mem_cgroup_shrink_node_zone+0x1c0/0x1c0
[ 288.310481] [<ffffffff8107a664>] kthread+0xe4/0x100
[ 288.310488] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
[ 288.310494] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
[ 288.310500] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
I've not seen any hang tasks just yet. The server seems to continue working. I will do more testing and get back to you with more info.
Andrei
From: "Ilya Dryomov" <ilya.dryomov@xxxxxxxxxxx>
To: "Andrei Mikhailovsky" <andrei@xxxxxxxxxx>
Cc: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
Sent: Friday, 28 November, 2014 5:28:28 PM
Subject: Re: Giant + nfs over cephfs hang tasks
On Fri, Nov 28, 2014 at 8:20 PM, Ilya Dryomov <ilya.dryomov@xxxxxxxxxxx> wrote:
> On Fri, Nov 28, 2014 at 8:19 PM, Ilya Dryomov <ilya.dryomov@xxxxxxxxxxx> wrote:
>> On Fri, Nov 28, 2014 at 8:13 PM, Andrei Mikhailovsky <andrei@xxxxxxxxxx> wrote:
>>> Ilya, yes I do! LIke these from different osds:
>>>
>>> [ 4422.212204] libceph: osd13 192.168.168.201:6819 socket closed (con state
>>> OPEN)
>>
>> Can you by any chance try a kernel from [1] ? It's based on Ubuntu
>> config and unless you are doing something fancy should boot your box.
>> You have to install it only on the client box of course.
>>
>> This may be related to the bug I'm currently trying to nail down and
>> I'd like to know if the latest bits make any difference.
>
> [1] http://gitbuilder.ceph.com/kernel-deb-precise-x86_64-basic/ref/testing/linux-image-3.18.0-rc6-ceph-00024-g72ca172_3.18.0-rc6-ceph-00024-g72ca172-1_amd64.deb
It's currently rebuilding because of an unrelated patch and will be
overwritten once gitbuilder is done. If it's not there by the time you
try use this link:
http://gitbuilder.ceph.com/kernel-deb-precise-x86_64-basic/sha1/72ca172a582d656930f413c3733401b8a5c120db/linux-image-3.18.0-rc6-ceph-00024-g72ca172_3.18.0-rc6-ceph-00024-g72ca172-1_amd64.deb
Thanks,
Ilya
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
