question about fuse livelock situation

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I found fuse livelock situation and report it for possibility of problem.

[Environment]
22.04 5.15.0-43-generic ubuntu kernel.
ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third
Generation NTFS Driver

[Problem]
I bumped on livelock and analyze it. and concluded that it is needed
to be fixed.
it happends when 3 operation concurrently progressing.

1) usb detach by user. and kernel detect it.
2) mount.ntfs umount request & device release operation
3) pool-udisksd umount operation.

[Conclusion]
1. mounted target device file must be released after /dev/fuse
release. it makes deadlocky scenario.
2. fuse file system "fuse_simple_request" should not be waiting
forever. it is hard to solve this situation by interrupting
application. huntask panic configuration make user kernel panic. user
don't know why.

[Analysis]
I got a kernel crash dump file. and analyze it.

here is the scenario description.
![1](https://user-images.githubusercontent.com/66734313/187164968-70880294-97bb-4ed9-807c-21d76b6f8f01.png)

1. kworker detect usb is detached from computer.
it is blocked by umount operation (pool-udiskd)
PID: 8      TASK: ffff88810029e400  CPU: 0   COMMAND: "kworker/0:1"
 #0 [ffffc9000006f6f0] __schedule at ffffffff81d57c0d
 #1 [ffffc9000006f778] schedule at ffffffff81d57fae
 #2 [ffffc9000006f798] rwsem_down_read_slowpath at ffffffff81d5a2de
 #3 [ffffc9000006f830] down_read at ffffffff81d5a373 // wait s_umount
 #4 [ffffc9000006f848] get_super at ffffffff813799ca
 #5 [ffffc9000006f878] fsync_bdev at ffffffff815c6f46
 #6 [ffffc9000006f8a0] delete_partition at ffffffff815e9328
 #7 [ffffc9000006f8c0] blk_drop_partitions at ffffffff815e9b3e
 #8 [ffffc9000006f8e8] del_gendisk at ffffffff815e75f1 // lock mutex
 #9 [ffffc9000006f930] sd_remove at ffffffff818cd325
#10 [ffffc9000006f958] __device_release_driver at ffffffff8184006f
#11 [ffffc9000006f990] device_release_driver at ffffffff818400a9
#12 [ffffc9000006f9b0] bus_remove_device at ffffffff8183f28e
#13 [ffffc9000006f9d8] device_del at ffffffff818399ac
#14 [ffffc9000006fa28] __scsi_remove_device at ffffffff818c2628
#15 [ffffc9000006fa50] scsi_forget_host at ffffffff818c029f
#16 [ffffc9000006fa70] scsi_remove_host at ffffffff818b3727
#17 [ffffc9000006fa98] usb_stor_disconnect at ffffffffc0659b20 [usb_storage]
#18 [ffffc9000006fac0] usb_unbind_interface at ffffffff8194ef30
#19 [ffffc9000006fb18] __device_release_driver at ffffffff8184006f
#20 [ffffc9000006fb50] device_release_driver at ffffffff818400a9
#21 [ffffc9000006fb70] bus_remove_device at ffffffff8183f28e
#22 [ffffc9000006fb98] device_del at ffffffff818399ac
#23 [ffffc9000006fbe8] usb_disable_device at ffffffff8194cdce
#24 [ffffc9000006fc30] usb_disconnect.cold at ffffffff81d19e09
#25 [ffffc9000006fc80] hub_port_connect at ffffffff81944bb8
#26 [ffffc9000006fd00] hub_port_connect_change at ffffffff819454b1
#27 [ffffc9000006fd70] port_event at ffffffff81945d77
#28 [ffffc9000006fe08] hub_event at ffffffff819460a7
#29 [ffffc9000006fe78] process_one_work at ffffffff810d9e7b
#30 [ffffc9000006fec8] worker_thread at ffffffff810da073
#31 [ffffc9000006ff10] kthread at ffffffff810e1cba
#32 [ffffc9000006ff50] ret_from_fork at ffffffff81004bc2


 #3 [ffffc9000006f830] down_read at ffffffff81d5a373
    ffffc9000006f838: [ffff888154d36800:kmalloc-2k] ffffc9000006f870
    ffffc9000006f848: get_super+154

2. it request umount. and release /dev/sdc file before release
/dev/fuse. but. it is blocked by usb detach.
PID: 18681  TASK: ffff88810e5b8000  CPU: 1   COMMAND: "mount.ntfs"
 #0 [ffffc90000ea7c50] __schedule at ffffffff81d57c0d
 #1 [ffffc90000ea7cd8] schedule at ffffffff81d57fae
 #2 [ffffc90000ea7cf8] schedule_preempt_disabled at ffffffff81d5839e
 #3 [ffffc90000ea7d08] __mutex_lock.constprop.0 at ffffffff81d59053
 #4 [ffffc90000ea7d80] __mutex_lock_slowpath at ffffffff81d59353
 #5 [ffffc90000ea7d90] mutex_lock at ffffffff81d59394 // wait open mutex
 #6 [ffffc90000ea7da8] blkdev_put at ffffffff815c765a
 #7 [ffffc90000ea7de0] blkdev_close at ffffffff815c86e7
 #8 [ffffc90000ea7df8] __fput at ffffffff8137704f
 #9 [ffffc90000ea7e30] ____fput at ffffffff8137724e
#10 [ffffc90000ea7e40] task_work_run at ffffffff810deb6d
#11 [ffffc90000ea7e68] exit_to_user_mode_loop at ffffffff81160fc0
#12 [ffffc90000ea7e90] exit_to_user_mode_prepare at ffffffff8116106c
#13 [ffffc90000ea7ea8] syscall_exit_to_user_mode at ffffffff81d4f327
#14 [ffffc90000ea7ec0] do_syscall_64 at ffffffff81d4b1b9
#15 [ffffc90000ea7ef8] exit_to_user_mode_prepare at ffffffff81161007
#16 [ffffc90000ea7f10] syscall_exit_to_user_mode at ffffffff81d4f327
#17 [ffffc90000ea7f28] do_syscall_64 at ffffffff81d4b1b9
    RIP: 00007f3e315fb117  RSP: 00007fff16bcb628  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 000055cc1b45c710  RCX: 00007f3e315fb117
    RDX: 0000000000000000  RSI: 0000000000000006  RDI: 0000000000000003
    RBP: 000055cc1b45c970   R8: 000055cc1b476940   R9: 0000000000000000
    R10: 0000000000000005  R11: 0000000000000246  R12: 00007f3e314e36c0
    R13: 000055cc1b45c710  R14: 000055cc1b467030  R15: 00000000ffffffff
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b


 #3 [ffffc90000ea7d08] __mutex_lock.constprop.0 at ffffffff81d59053
    ffffc90000ea7d10: [ffff8881caad1880:kmalloc-512] 00000002547b2ff8
    ffffc90000ea7d20: [ffff8881caad1880:kmalloc-512]
[ffff8881caad1880:kmalloc-512]
    ffffc90000ea7d30: [ffff88810e5b8000:task_struct] 0000000008000008
    ffffc90000ea7d40: ffffc90000ea7de0 fcee7cb500545100
    ffffc90000ea7d50: [ffff888110f99f40:bdev_cache]
[ffff8881caad1870:kmalloc-512]
    ffffc90000ea7d60: [ffff8881caad1800:kmalloc-512]
[ffff8881caad1870:kmalloc-512]
    ffffc90000ea7d70: [ffff8881bcd4ccc0:dentry] ffffc90000ea7d88
    ffffc90000ea7d80: __mutex_lock_slowpath+19
 #4 [ffffc90000ea7d80] __mutex_lock_slowpath at ffffff

3. we can know that it is releasing /dev/sdc file and not release /dev/fuse.
crash> files 18681
PID: 18681  TASK: ffff88810e5b8000  CPU: 1   COMMAND: "mount.ntfs"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  1 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  2 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  4 ffff888154001f00 ffff8881020cce40 ffff888100f90f50 CHR  /dev/fuse
  5 ffff888102f74200 ffff8881c2c5dd80 ffff88815db73e40 SOCK UNIX
  8 ffff888106412800 ffff888104785540 ffff888104a9e470 REG  /proc/804/mountinfo
  9 ffff888106412100 ffff888100443840 ffff8881022a9688 REG  /proc/swaps
 14 ffff88810a720700 ffff888104785540 ffff888104a9e470 REG  /proc/804/mountinfo

4. pool udisksd is progressing umount process (cuz uhelper=udisksd2)
but it is waiting forever cuz mount.ntfs does not progress. (if it is
to be released, /dev/fuse
must be released. but mount.ntfs blocked by kworker)

PI3: 18741  TASK: ffff8881a702b200  CPU: 1   COMMAND: "pool-udisksd"
 #0 [ffffc900022a3be8] __schedule at ffffffff81d57c0d
 #1 [ffffc900022a3c70] schedule at ffffffff81d57fae
 #2 [ffffc900022a3c90] request_wait_answer at ffffffff814ea190
 #3 [ffffc900022a3cf0] fuse_simple_request at ffffffff814ec9c1
 #4 [ffffc900022a3d30] fuse_send_destroy at ffffffff814f743f
 #5 [ffffc900022a3db8] fuse_conn_destroy at ffffffff814f7622
 #6 [ffffc900022a3dd0] fuse_kill_sb_blk at ffffffff814f7d82
 #7 [ffffc900022a3e00] deactivate_locked_super at ffffffff8137863b
 #8 [ffffc900022a3e20] deactivate_super at ffffffff813786f0 // lock s_umount
 #9 [ffffc900022a3e38] cleanup_mnt at ffffffff8139f350
#10 [ffffc900022a3e78] __cleanup_mnt at ffffffff8139f422
#11 [ffffc900022a3e88] task_work_run at ffffffff810deb6d
#12 [ffffc900022a3eb0] exit_to_user_mode_loop at ffffffff81160fc0
#13 [ffffc900022a3ed8] exit_to_user_mode_prepare at ffffffff8116106c
#14 [ffffc900022a3ef0] syscall_exit_to_user_mode at ffffffff81d4f327
#15 [ffffc900022a3f08] do_syscall_64 at ffffffff81d4b1b9
#16 [ffffc900022a3f50] entry_SYSCALL_64_after_hwframe at ffffffff81e0007c
    RIP: 00007f110d17bbeb  RSP: 00007f110beb63b8  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 00007f10fc0257e0  RCX: 00007f110d17bbeb
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 00007f10fc012bb0
    RBP: 00007f10fc09f480   R8: 0000000000000000   R9: 00007f10fc05edf0
    R10: 00007f10fc05ee30  R11: 0000000000000246  R12: 0000000000000000
    R13: 00007f10fc012bb0  R14: 00007f10fc09f590  R15: 0000000000000000
    ORIG_RAX: 00000000000000a6  CS: 0033  SS: 002b



[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [NTFS 3]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [NTFS 3]     [Samba]     [Device Mapper]     [CEPH Development]

  Powered by Linux