Hi ceph-devel,
I know this mail might be better to send to fuse-devel, but I haven't
been approved by fuse-devel yet since I subscribed it.
Occasionally we met ceph-fuse hanging issue when mapping ceph-fuse
directory into docker container using aufs.
Here is the dmesg:
[809401.613923] aufs au_opts_verify:1602:docker[36838]: dirperm1
breaks the protection by the permission bits on the lower branch
[825359.968412] aufs au_opts_verify:1602:docker[32013]: dirperm1
breaks the protection by the permission bits on the lower branch
[825359.970719] aufs au_opts_verify:1602:docker[32013]: dirperm1
breaks the protection by the permission bits on the lower branch
[825359.973689] aufs au_opts_verify:1602:docker[44954]: dirperm1
breaks the protection by the permission bits on the lower branch
[836447.630952] INFO: task df:30614 blocked for more than 120 seconds.
[836447.630955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[836447.630957] df D ffff88203f851e00 0 30614 1 0x00000004
[836447.630960] ffff880a27b6fd08 0000000000000002 ffff880a27b6ffd8
0000000000011e00
[836447.630964] ffff880a27b6ffd8 0000000000011e00 ffff880fe93f4b60
ffff880009f98000
[836447.630967] ffff881074050320 ffff881fd6333000 ffff880a27b6fd30
ffff881074050400
[836447.630970] Call Trace:
[836447.630977] [<ffffffff81acb689>] schedule+0x29/0x70
[836447.630983] [<ffffffff812be96d>] __fuse_request_send+0xdd/0x290
[836447.630987] [<ffffffff81066150>] ? wake_up_bit+0x30/0x30
[836447.630989] [<ffffffff812beb32>] fuse_request_send+0x12/0x20
[836447.630992] [<ffffffff812c3859>] fuse_do_getattr+0x109/0x2a0
[836447.630995] [<ffffffff812c4cd5>] fuse_update_attributes+0x75/0x80
[836447.630997] [<ffffffff812c4d23>] fuse_getattr+0x43/0x50
[836447.631001] [<ffffffff81168249>] vfs_getattr+0x29/0x40
[836447.631002] [<ffffffff811683b2>] vfs_fstatat+0x62/0xa0
[836447.631004] [<ffffffff8116859f>] SYSC_newstat+0x1f/0x40
[836447.631009] [<ffffffff8100e4f8>] ? syscall_trace_enter+0x18/0x210
[836447.631012] [<ffffffff81ad50bc>] ? tracesys+0x7e/0xe2
[836447.631014] [<ffffffff811689ee>] SyS_newstat+0xe/0x10
[836447.631016] [<ffffffff81ad511b>] tracesys+0xdd/0xe2
>From above msg, it seems to be hung at __fuse_request_send, which will
queue request and wait for reply from ceph-fuse. When this happens,
'ls' ceph-fuse directory or 'df' outside docker container will also
get hang.
I generated ceph-fuse's core dump and found client_lock was not held
by any thread. So I wonder if something wrong with ceph-fuse that
can't get the request from the fuse's queue? Or something else related
to fuse itself?
Thanks.
Regards,
Zhi Zhang (David)
Contact: zhang.david2011@xxxxxxxxx
zhangz.david@xxxxxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
