I was trying some experiments on 5.17-rc4 with lockdep enabled.
And I noticed this deadlock warning.
It's about a circular dependency locking fs_reclaim lock with srv_mutex held.
Does someone here understand this dependency?
--
Regards,
Shyam
[Mon Feb 14 16:31:14 2022] ======================================================
[Mon Feb 14 16:31:14 2022] WARNING: possible circular locking dependency detected
[Mon Feb 14 16:31:14 2022] 5.17.0-rc3+ #4 Tainted: G OE
[Mon Feb 14 16:31:14 2022] ------------------------------------------------------
[Mon Feb 14 16:31:14 2022] kswapd0/48 is trying to acquire lock:
[Mon Feb 14 16:31:14 2022] ffff888113aca2e0 (&tcp_ses->srv_mutex){+.+.}-{3:3}, at: compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022]
but task is already holding lock:
[Mon Feb 14 16:31:14 2022] ffffffffae4162a0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0xa36/0xb20
[Mon Feb 14 16:31:14 2022]
which lock already depends on the new lock.
[Mon Feb 14 16:31:14 2022]
the existing dependency chain (in reverse order) is:
[Mon Feb 14 16:31:14 2022]
-> #1 (fs_reclaim){+.+.}-{0:0}:
[Mon Feb 14 16:31:14 2022] fs_reclaim_acquire+0xc2/0xf0
[Mon Feb 14 16:31:14 2022] __kmalloc_node+0x61/0x4a0
[Mon Feb 14 16:31:14 2022] crypto_create_tfm_node+0x53/0x180
[Mon Feb 14 16:31:14 2022] crypto_alloc_tfm_node+0x98/0x130
[Mon Feb 14 16:31:14 2022] crypto_alloc_shash+0x1f/0x30
[Mon Feb 14 16:31:14 2022] cifs_alloc_hash+0x46/0x140 [cifs]
[Mon Feb 14 16:31:14 2022] smb311_crypto_shash_allocate+0x39/0xc0 [cifs]
[Mon Feb 14 16:31:14 2022] smb311_update_preauth_hash+0xb5/0x240 [cifs]
[Mon Feb 14 16:31:14 2022] compound_send_recv+0xdc2/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_send_recv+0x22/0x30 [cifs]
[Mon Feb 14 16:31:14 2022] SMB2_negotiate+0x56d/0x1a40 [cifs]
[Mon Feb 14 16:31:14 2022] smb2_negotiate+0x57/0x70 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_negotiate_protocol+0x10b/0x180 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_get_smb_ses+0x627/0x10d0 [cifs]
[Mon Feb 14 16:31:14 2022] mount_get_conns+0x8c/0x660 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_mount+0xf8/0xe20 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_smb3_do_mount+0x1dd/0xbe0 [cifs]
[Mon Feb 14 16:31:14 2022] smb3_get_tree+0x1a0/0x2e0 [cifs]
[Mon Feb 14 16:31:14 2022] vfs_get_tree+0x52/0x140
[Mon Feb 14 16:31:14 2022] path_mount+0x635/0x10c0
[Mon Feb 14 16:31:14 2022] __x64_sys_mount+0x1bf/0x210
[Mon Feb 14 16:31:14 2022] do_syscall_64+0x5c/0xc0
[Mon Feb 14 16:31:14 2022] entry_SYSCALL_64_after_hwframe+0x44/0xae
[Mon Feb 14 16:31:14 2022]
-> #0 (&tcp_ses->srv_mutex){+.+.}-{3:3}:
[Mon Feb 14 16:31:14 2022] __lock_acquire+0x1c7a/0x30a0
[Mon Feb 14 16:31:14 2022] lock_acquire+0x18c/0x430
[Mon Feb 14 16:31:14 2022] __mutex_lock+0x11f/0xc70
[Mon Feb 14 16:31:14 2022] mutex_lock_nested+0x1b/0x20
[Mon Feb 14 16:31:14 2022] compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_send_recv+0x22/0x30 [cifs]
[Mon Feb 14 16:31:14 2022] SMB2_write+0x45f/0x6e0 [cifs]
[Mon Feb 14 16:31:14 2022] smb2_sync_write+0x7e/0x90 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_write+0x2cf/0x7b0 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_writepage_locked+0x59c/0x730 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_writepage+0x15/0x30 [cifs]
[Mon Feb 14 16:31:14 2022] pageout+0x23b/0x650
[Mon Feb 14 16:31:14 2022] shrink_page_list+0x152d/0x19f0
[Mon Feb 14 16:31:14 2022] shrink_lruvec+0xa89/0x14d0
[Mon Feb 14 16:31:14 2022] shrink_node+0x518/0xd40
[Mon Feb 14 16:31:14 2022] balance_pgdat+0x58c/0xb20
[Mon Feb 14 16:31:14 2022] kswapd+0x40f/0x7b0
[Mon Feb 14 16:31:14 2022] kthread+0x174/0x1b0
[Mon Feb 14 16:31:14 2022] ret_from_fork+0x22/0x30
[Mon Feb 14 16:31:14 2022]
other info that might help us debug this:
[Mon Feb 14 16:31:14 2022] Possible unsafe locking scenario:
[Mon Feb 14 16:31:14 2022] CPU0 CPU1
[Mon Feb 14 16:31:14 2022] ---- ----
[Mon Feb 14 16:31:14 2022] lock(fs_reclaim);
[Mon Feb 14 16:31:14 2022] lock(&tcp_ses->srv_mutex);
[Mon Feb 14 16:31:14 2022] lock(fs_reclaim);
[Mon Feb 14 16:31:14 2022] lock(&tcp_ses->srv_mutex);
[Mon Feb 14 16:31:14 2022]
*** DEADLOCK ***
[Mon Feb 14 16:31:14 2022] 1 lock held by kswapd0/48:
[Mon Feb 14 16:31:14 2022] #0: ffffffffae4162a0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0xa36/0xb20
[Mon Feb 14 16:31:14 2022]
stack backtrace:
[Mon Feb 14 16:31:14 2022] CPU: 0 PID: 48 Comm: kswapd0 Tainted: G OE 5.17.0-rc3+ #4
[Mon Feb 14 16:31:14 2022] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.0 12/17/2019
[Mon Feb 14 16:31:14 2022] Call Trace:
[Mon Feb 14 16:31:14 2022] <TASK>
[Mon Feb 14 16:31:14 2022] dump_stack_lvl+0x5d/0x78
[Mon Feb 14 16:31:14 2022] dump_stack+0x10/0x12
[Mon Feb 14 16:31:14 2022] print_circular_bug.cold+0x146/0x14b
[Mon Feb 14 16:31:14 2022] check_noncircular+0x1e2/0x210
[Mon Feb 14 16:31:14 2022] ? print_circular_bug+0x120/0x120
[Mon Feb 14 16:31:14 2022] ? __kasan_check_read+0x11/0x20
[Mon Feb 14 16:31:14 2022] ? call_rcu_zapped+0x90/0x90
[Mon Feb 14 16:31:14 2022] ? lock_chain_count+0x20/0x20
[Mon Feb 14 16:31:14 2022] __lock_acquire+0x1c7a/0x30a0
[Mon Feb 14 16:31:14 2022] ? lockdep_hardirqs_on_prepare+0x230/0x230
[Mon Feb 14 16:31:14 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:14 2022] ? lock_is_held_type+0xea/0x140
[Mon Feb 14 16:31:14 2022] lock_acquire+0x18c/0x430
[Mon Feb 14 16:31:14 2022] ? compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] ? lock_release+0x490/0x490
[Mon Feb 14 16:31:14 2022] ? lock_release+0x28e/0x490
[Mon Feb 14 16:31:14 2022] ? wait_for_free_credits+0x73f/0xaa0 [cifs]
[Mon Feb 14 16:31:14 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:14 2022] ? lock_is_held_type+0xea/0x140
[Mon Feb 14 16:31:14 2022] __mutex_lock+0x11f/0xc70
[Mon Feb 14 16:31:14 2022] ? compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] ? compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] ? wait_for_free_credits+0x793/0xaa0 [cifs]
[Mon Feb 14 16:31:14 2022] ? mutex_lock_io_nested+0xbf0/0xbf0
[Mon Feb 14 16:31:14 2022] ? lock_release+0x28e/0x490
[Mon Feb 14 16:31:14 2022] ? compound_send_recv+0x239/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] ? trace_smb3_credit_timeout+0x170/0x170 [cifs]
[Mon Feb 14 16:31:14 2022] ? lock_downgrade+0x3f0/0x3f0
[Mon Feb 14 16:31:14 2022] ? rwlock_bug.part.0+0x60/0x60
[Mon Feb 14 16:31:14 2022] ? lockdep_unlock+0xd3/0x120
[Mon Feb 14 16:31:14 2022] mutex_lock_nested+0x1b/0x20
[Mon Feb 14 16:31:14 2022] ? mutex_lock_nested+0x1b/0x20
[Mon Feb 14 16:31:14 2022] compound_send_recv+0x2e8/0x12c0 [cifs]
[Mon Feb 14 16:31:14 2022] ? cifs_pick_channel+0xe0/0xe0 [cifs]
[Mon Feb 14 16:31:14 2022] ? find_held_lock+0x8e/0xa0
[Mon Feb 14 16:31:14 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:14 2022] ? lock_release+0x28e/0x490
[Mon Feb 14 16:31:14 2022] ? lock_downgrade+0x3f0/0x3f0
[Mon Feb 14 16:31:14 2022] ? rwlock_bug.part.0+0x60/0x60
[Mon Feb 14 16:31:14 2022] cifs_send_recv+0x22/0x30 [cifs]
[Mon Feb 14 16:31:14 2022] SMB2_write+0x45f/0x6e0 [cifs]
[Mon Feb 14 16:31:14 2022] ? smb2_async_writev+0xd60/0xd60 [cifs]
[Mon Feb 14 16:31:14 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:14 2022] ? lock_release+0x28e/0x490
[Mon Feb 14 16:31:14 2022] ? _get_xid+0x96/0xe0 [cifs]
[Mon Feb 14 16:31:14 2022] ? rwlock_bug.part.0+0x60/0x60
[Mon Feb 14 16:31:14 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:14 2022] ? lock_is_held_type+0xea/0x140
[Mon Feb 14 16:31:14 2022] smb2_sync_write+0x7e/0x90 [cifs]
[Mon Feb 14 16:31:14 2022] cifs_write+0x2cf/0x7b0 [cifs]
[Mon Feb 14 16:31:14 2022] ? __cifs_writev+0x6a0/0x6a0 [cifs]
[Mon Feb 14 16:31:15 2022] ? _raw_spin_unlock+0x2d/0x50
[Mon Feb 14 16:31:15 2022] cifs_writepage_locked+0x59c/0x730 [cifs]
[Mon Feb 14 16:31:15 2022] ? cifs_writepages+0x1540/0x1540 [cifs]
[Mon Feb 14 16:31:15 2022] ? __rcu_read_unlock+0x7c/0x2e0
[Mon Feb 14 16:31:15 2022] ? folio_clear_dirty_for_io+0x144/0x300
[Mon Feb 14 16:31:15 2022] cifs_writepage+0x15/0x30 [cifs]
[Mon Feb 14 16:31:15 2022] pageout+0x23b/0x650
[Mon Feb 14 16:31:15 2022] ? check_move_unevictable_pages+0x850/0x850
[Mon Feb 14 16:31:15 2022] ? __kasan_check_read+0x11/0x20
[Mon Feb 14 16:31:15 2022] ? folio_mapped+0x9b/0x130
[Mon Feb 14 16:31:15 2022] shrink_page_list+0x152d/0x19f0
[Mon Feb 14 16:31:15 2022] ? pageout+0x650/0x650
[Mon Feb 14 16:31:15 2022] ? __this_cpu_preempt_check+0x11/0x20
[Mon Feb 14 16:31:15 2022] ? lock_release+0x240/0x490
[Mon Feb 14 16:31:15 2022] ? shrink_lruvec+0xa54/0x14d0
[Mon Feb 14 16:31:15 2022] ? mark_held_locks+0x6b/0x90
[Mon Feb 14 16:31:15 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:15 2022] ? _raw_spin_unlock_irq+0x28/0x60
[Mon Feb 14 16:31:15 2022] ? trace_hardirqs_on+0x2b/0x110
[Mon Feb 14 16:31:15 2022] shrink_lruvec+0xa89/0x14d0
[Mon Feb 14 16:31:15 2022] ? shrink_active_list+0x980/0x980
[Mon Feb 14 16:31:15 2022] ? lock_is_held_type+0xea/0x140
[Mon Feb 14 16:31:15 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:15 2022] ? percpu_ref_put_many.constprop.0+0x5c/0x140
[Mon Feb 14 16:31:15 2022] ? lock_downgrade+0x3f0/0x3f0
[Mon Feb 14 16:31:15 2022] ? lock_downgrade+0x3f0/0x3f0
[Mon Feb 14 16:31:15 2022] ? lock_downgrade+0x3f0/0x3f0
[Mon Feb 14 16:31:15 2022] ? __rcu_read_unlock+0x7c/0x2e0
[Mon Feb 14 16:31:15 2022] ? __kasan_check_read+0x11/0x20
[Mon Feb 14 16:31:15 2022] ? mem_cgroup_calculate_protection+0x204/0x2a0
[Mon Feb 14 16:31:15 2022] shrink_node+0x518/0xd40
[Mon Feb 14 16:31:15 2022] balance_pgdat+0x58c/0xb20
[Mon Feb 14 16:31:15 2022] ? do_try_to_free_pages+0xad0/0xad0
[Mon Feb 14 16:31:15 2022] ? find_held_lock+0x8e/0xa0
[Mon Feb 14 16:31:15 2022] ? lock_release+0x28e/0x490
[Mon Feb 14 16:31:15 2022] ? finish_wait+0xa6/0xd0
[Mon Feb 14 16:31:15 2022] ? lockdep_hardirqs_on+0x83/0x110
[Mon Feb 14 16:31:15 2022] ? _raw_spin_unlock_irqrestore+0x31/0x60
[Mon Feb 14 16:31:15 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:15 2022] ? lock_is_held_type+0xea/0x140
[Mon Feb 14 16:31:15 2022] kswapd+0x40f/0x7b0
[Mon Feb 14 16:31:15 2022] ? balance_pgdat+0xb20/0xb20
[Mon Feb 14 16:31:15 2022] ? __this_cpu_preempt_check+0x13/0x20
[Mon Feb 14 16:31:15 2022] ? lockdep_hardirqs_on+0x83/0x110
[Mon Feb 14 16:31:15 2022] ? wait_woken+0xb0/0xb0
[Mon Feb 14 16:31:15 2022] ? preempt_count_sub+0x18/0xc0
[Mon Feb 14 16:31:15 2022] ? __kasan_check_read+0x11/0x20
[Mon Feb 14 16:31:15 2022] ? balance_pgdat+0xb20/0xb20
[Mon Feb 14 16:31:15 2022] kthread+0x174/0x1b0
[Mon Feb 14 16:31:15 2022] ? kthread_complete_and_exit+0x30/0x30
[Mon Feb 14 16:31:15 2022] ret_from_fork+0x22/0x30
[Mon Feb 14 16:31:15 2022] </TASK>
