Hello
I found this issue from v6.9-rc3, here is the reproducer and dmesg
log[1][2], and I only see one xfs commit merged from
v6.9-rc3(e23d7e82b707 xfs: allow cross-linking special files without
project quota), please help check it and let me know if you need any
info/test for it, thanks.
[1] Reproducer:
mkfs.xfs -f /dev/nvme0n1p1
mkfs.xfs -f /dev/nvme1n1p1
mount /dev/nvme0n1p1 /mnt/fortest/nvme0n1p1
mount /dev/nvme1n1p1 /mnt/fortest/nvme1n1p1
iozone -t 6 -F /mnt/fortest/nvme0n1p1/iozone0.tmp
/mnt/fortest/nvme0n1p1/iozone1.tmp /mnt/fortest/nvme0n1p1/iozone2.tmp
/mnt/fortest/nvme0n1p1/iozone3.tmp /mnt/fortest/nvme0n1p1/iozone4.tmp
/mnt/fortest/nvme0n1p1/iozone5.tmp -s 102400M -r 1M -i 0 -i 1 -+d -+n
-+u -x -e -w -C
umount -l /dev/nvme0n1p1
iozone -t 6 -F /mnt/fortest/nvme1n1p1/iozone0.tmp
/mnt/fortest/nvme1n1p1/iozone1.tmp /mnt/fortest/nvme1n1p1/iozone2.tmp
/mnt/fortest/nvme1n1p1/iozone3.tmp /mnt/fortest/nvme1n1p1/iozone4.tmp
/mnt/fortest/nvme1n1p1/iozone5.tmp -s 102400M -r 1M -i 0 -i 1 -+d -+n
-+u -x -e -w -C
umount -l /dev/nvme1n1p1
[2] dmesg:
[ 1858.395989] nvme0n1:
[ 1858.455944] nvme0n1:
[ 1858.632024] nvme1n1:
[ 1858.661671] nvme1n1:
[ 1858.933588] nvme0n1: p1
[ 1864.105590] nvme1n1: p1
[ 1877.982176] EXT4-fs (nvme1n1p1): mounted filesystem
04118d2e-23c3-4348-a64f-477bdbfb84fe r/w with ordered data mode. Quota
mode: none.
[ 1878.512723] EXT4-fs (nvme0n1p1): mounted filesystem
c8fdab45-48a3-438d-b8b2-656164050649 r/w with ordered data mode. Quota
mode: none.
[ 3745.664603] perf: interrupt took too long (3139 > 3137), lowering
kernel.perf_event_max_sample_rate to 63000
[ 9648.516436] EXT4-fs (nvme1n1p1): unmounting filesystem
04118d2e-23c3-4348-a64f-477bdbfb84fe.
[ 9745.093719] EXT4-fs (nvme0n1p1): unmounting filesystem
c8fdab45-48a3-438d-b8b2-656164050649.
[ 9750.130878] XFS (nvme0n1p1): Mounting V5 Filesystem
69ef6a54-8c23-4e26-8381-3072b0bcc9ef
[ 9750.161564] XFS (nvme0n1p1): Ending clean mount
[ 9750.839996] ======================================================
[ 9750.846181] WARNING: possible circular locking dependency detected
[ 9750.852375] 6.9.0 #1 Not tainted
[ 9750.855636] ------------------------------------------------------
[ 9750.861840] kswapd0/197 is trying to acquire lock:
[ 9750.866644] ffff8882093cf920 (&xfs_nondir_ilock_class){++++}-{3:3},
at: xfs_can_free_eofblocks+0x300/0x570 [xfs]
[ 9750.877178]
but task is already holding lock:
[ 9750.883017] ffffffff8af48f60 (fs_reclaim){+.+.}-{0:0}, at:
balance_pgdat+0x5a4/0x1010
[ 9750.890867]
which lock already depends on the new lock.
[ 9750.899039]
the existing dependency chain (in reverse order) is:
[ 9750.906523]
-> #1 (fs_reclaim){+.+.}-{0:0}:
[ 9750.912207] __lock_acquire+0xbd9/0x1c00
[ 9750.916669] lock_acquire+0x1d9/0x600
[ 9750.920864] fs_reclaim_acquire+0x103/0x160
[ 9750.925579] __kmalloc+0x9d/0x480
[ 9750.929424] xfs_attr_shortform_list+0x576/0x14f0 [xfs]
[ 9750.935555] xfs_attr_list+0x1ca/0x260 [xfs]
[ 9750.940701] xfs_vn_listxattr+0xe9/0x160 [xfs]
[ 9750.946040] listxattr+0x5a/0xf0
[ 9750.949810] __x64_sys_flistxattr+0x124/0x1b0
[ 9750.954696] do_syscall_64+0x8f/0x180
[ 9750.958892] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 9750.964483]
-> #0 (&xfs_nondir_ilock_class){++++}-{3:3}:
[ 9750.971292] check_prev_add+0x1b7/0x2360
[ 9750.975745] validate_chain+0xa42/0xe00
[ 9750.980106] __lock_acquire+0xbd9/0x1c00
[ 9750.984550] lock_acquire+0x1d9/0x600
[ 9750.988737] down_read_nested+0xa5/0x4d0
[ 9750.993182] xfs_can_free_eofblocks+0x300/0x570 [xfs]
[ 9750.999059] xfs_inode_mark_reclaimable+0x197/0x230 [xfs]
[ 9751.005247] destroy_inode+0xbc/0x1a0
[ 9751.009442] dispose_list+0xf5/0x1b0
[ 9751.013549] prune_icache_sb+0xe0/0x150
[ 9751.017908] super_cache_scan+0x30f/0x4e0
[ 9751.022449] do_shrink_slab+0x382/0xe30
[ 9751.026817] shrink_slab_memcg+0x45a/0x900
[ 9751.031438] shrink_slab+0x3f9/0x4e0
[ 9751.035535] shrink_one+0x3ff/0x6d0
[ 9751.039557] shrink_many+0x2ed/0xc60
[ 9751.043657] lru_gen_shrink_node+0x43b/0x5a0
[ 9751.048450] balance_pgdat+0x4e9/0x1010
[ 9751.052807] kswapd+0x3ae/0x660
[ 9751.056475] kthread+0x2f6/0x3e0
[ 9751.060235] ret_from_fork+0x30/0x70
[ 9751.064344] ret_from_fork_asm+0x1a/0x30
[ 9751.068799]
other info that might help us debug this:
[ 9751.076799] Possible unsafe locking scenario:
[ 9751.082714] CPU0 CPU1
[ 9751.087248] ---- ----
[ 9751.091780] lock(fs_reclaim);
[ 9751.094935] lock(&xfs_nondir_ilock_class);
[ 9751.101730] lock(fs_reclaim);
[ 9751.107398] rlock(&xfs_nondir_ilock_class);
[ 9751.111766]
*** DEADLOCK ***
[ 9751.117684] 2 locks held by kswapd0/197:
[ 9751.121610] #0: ffffffff8af48f60 (fs_reclaim){+.+.}-{0:0}, at:
balance_pgdat+0x5a4/0x1010
[ 9751.129888] #1: ffff88822f2280e8
(&type->s_umount_key#58){++++}-{3:3}, at: super_cache_scan+0x7e/0x4e0
[ 9751.139298]
stack backtrace:
[ 9751.143660] CPU: 12 PID: 197 Comm: kswapd0 Kdump: loaded Not tainted 6.9.0 #1
[ 9751.150800] Hardware name: Dell Inc. PowerEdge R6515/07PXPY, BIOS
2.13.3 09/12/2023
[ 9751.158453] Call Trace:
[ 9751.160907] <TASK>
[ 9751.163013] dump_stack_lvl+0x7e/0xc0
[ 9751.166687] check_noncircular+0x2f6/0x3e0
[ 9751.170797] ? __pfx_check_noncircular+0x10/0x10
[ 9751.175424] ? check_irq_usage+0x19f/0xa60
[ 9751.179528] ? __bfs+0x247/0x5e0
[ 9751.182762] ? __pfx_hlock_conflict+0x10/0x10
[ 9751.187130] ? srso_return_thunk+0x5/0x5f
[ 9751.191167] check_prev_add+0x1b7/0x2360
[ 9751.195098] ? kernel_text_address+0x13/0xd0
[ 9751.199380] validate_chain+0xa42/0xe00
[ 9751.203234] ? __pfx_validate_chain+0x10/0x10
[ 9751.207599] ? srso_return_thunk+0x5/0x5f
[ 9751.211613] ? mark_lock.part.0+0x77/0x880
[ 9751.215728] __lock_acquire+0xbd9/0x1c00
[ 9751.219665] ? srso_return_thunk+0x5/0x5f
[ 9751.223687] lock_acquire+0x1d9/0x600
[ 9751.227362] ? xfs_can_free_eofblocks+0x300/0x570 [xfs]
[ 9751.232868] ? __pfx_lock_acquire+0x10/0x10
[ 9751.237057] ? srso_return_thunk+0x5/0x5f
[ 9751.241068] ? mark_lock.part.0+0x77/0x880
[ 9751.245173] ? srso_return_thunk+0x5/0x5f
[ 9751.249195] ? srso_return_thunk+0x5/0x5f
[ 9751.253217] down_read_nested+0xa5/0x4d0
[ 9751.257146] ? xfs_can_free_eofblocks+0x300/0x570 [xfs]
[ 9751.262653] ? __pfx_down_read_nested+0x10/0x10
[ 9751.267193] ? srso_return_thunk+0x5/0x5f
[ 9751.271210] ? rcu_is_watching+0x11/0xb0
[ 9751.275149] ? srso_return_thunk+0x5/0x5f
[ 9751.279176] xfs_can_free_eofblocks+0x300/0x570 [xfs]
[ 9751.284602] ? __pfx___lock_release+0x10/0x10
[ 9751.288974] ? __pfx_xfs_can_free_eofblocks+0x10/0x10 [xfs]
[ 9751.294912] ? __pfx_do_raw_spin_trylock+0x10/0x10
[ 9751.299722] ? xfs_inode_mark_reclaimable+0x18b/0x230 [xfs]
[ 9751.305672] ? srso_return_thunk+0x5/0x5f
[ 9751.309696] xfs_inode_mark_reclaimable+0x197/0x230 [xfs]
[ 9751.315372] destroy_inode+0xbc/0x1a0
[ 9751.319052] dispose_list+0xf5/0x1b0
[ 9751.322645] prune_icache_sb+0xe0/0x150
[ 9751.326488] ? __pfx_prune_icache_sb+0x10/0x10
[ 9751.330945] ? srso_return_thunk+0x5/0x5f
[ 9751.334966] super_cache_scan+0x30f/0x4e0
[ 9751.338990] do_shrink_slab+0x382/0xe30
[ 9751.342844] shrink_slab_memcg+0x45a/0x900
[ 9751.346952] ? shrink_slab_memcg+0x13e/0x900
[ 9751.351234] ? __pfx_shrink_slab_memcg+0x10/0x10
[ 9751.355857] ? srso_return_thunk+0x5/0x5f
[ 9751.359877] ? mem_cgroup_get_nr_swap_pages+0x98/0x120
[ 9751.365031] ? srso_return_thunk+0x5/0x5f
[ 9751.369045] ? try_to_shrink_lruvec+0x5aa/0x820
[ 9751.373594] shrink_slab+0x3f9/0x4e0
[ 9751.377180] ? __pfx_shrink_slab+0x10/0x10
[ 9751.381282] ? local_clock_noinstr+0x9/0xc0
[ 9751.385471] ? srso_return_thunk+0x5/0x5f
[ 9751.389489] ? __lock_release+0x486/0x960
[ 9751.393504] ? __pfx_try_to_shrink_lruvec+0x10/0x10
[ 9751.398393] ? srso_return_thunk+0x5/0x5f
[ 9751.402414] ? srso_return_thunk+0x5/0x5f
[ 9751.406432] ? mem_cgroup_calculate_protection+0x27d/0x500
[ 9751.411924] shrink_one+0x3ff/0x6d0
[ 9751.415427] shrink_many+0x2ed/0xc60
[ 9751.419008] ? shrink_many+0x2d6/0xc60
[ 9751.422767] ? srso_return_thunk+0x5/0x5f
[ 9751.426787] lru_gen_shrink_node+0x43b/0x5a0
[ 9751.431068] ? __pfx_lru_gen_shrink_node+0x10/0x10
[ 9751.435873] ? srso_return_thunk+0x5/0x5f
[ 9751.439885] ? pgdat_balanced+0xbb/0x1a0
[ 9751.443817] balance_pgdat+0x4e9/0x1010
[ 9751.447670] ? __pfx_balance_pgdat+0x10/0x10
[ 9751.451954] ? srso_return_thunk+0x5/0x5f
[ 9751.455971] ? srso_return_thunk+0x5/0x5f
[ 9751.459982] ? set_pgdat_percpu_threshold+0x1be/0x2a0
[ 9751.465065] ? srso_return_thunk+0x5/0x5f
[ 9751.469083] ? srso_return_thunk+0x5/0x5f
[ 9751.473096] ? preempt_schedule_common+0x44/0xb0
[ 9751.477731] kswapd+0x3ae/0x660
[ 9751.480898] ? __pfx_kswapd+0x10/0x10
[ 9751.484570] ? srso_return_thunk+0x5/0x5f
[ 9751.488594] ? srso_return_thunk+0x5/0x5f
[ 9751.492614] ? __kthread_parkme+0xc4/0x200
[ 9751.496725] ? __pfx_kswapd+0x10/0x10
[ 9751.500400] kthread+0x2f6/0x3e0
[ 9751.503646] ? _raw_spin_unlock_irq+0x24/0x50
[ 9751.508013] ? __pfx_kthread+0x10/0x10
[ 9751.511779] ret_from_fork+0x30/0x70
[ 9751.515362] ? __pfx_kthread+0x10/0x10
[ 9751.519117] ret_from_fork_asm+0x1a/0x30
[ 9751.523064] </TASK>
[ 9751.956668] XFS (nvme1n1p1): Mounting V5 Filesystem
46bda624-b0a2-47f4-b8db-ec072b19a662
[ 9752.086777] XFS (nvme1n1p1): Ending clean mount
[11490.037665] XFS (nvme1n1p1): Unmounting Filesystem
46bda624-b0a2-47f4-b8db-ec072b19a662
[11585.770356] XFS (nvme0n1p1): Unmounting Filesystem
69ef6a54-8c23-4e26-8381-3072b0bcc9ef
Thanks
Yi
