On Mon, May 14, 2018 at 11:51 AM, Yan, Zheng <zyan@xxxxxxxxxx> wrote:
>
> [...]
>
> Sorry, I was wrong about where he hang occurs. It’s at
>
> [<0>] io_schedule+0xd/0x30
> [<0>] wait_on_page_bit_common+0xc6/0x130
> [<0>] __filemap_fdatawait_range+0xbd/0x100
> [<0>] filemap_fdatawait_keep_errors+0x15/0x40
> [<0>] sync_inodes_sb+0x1cf/0x240
> [<0>] sync_filesystem+0x52/0x90
> [<0>] generic_shutdown_super+0x1d/0x110
> [<0>] ceph_kill_sb+0x28/0x80 [ceph]
> [<0>] deactivate_locked_super+0x35/0x60
> [<0>] cleanup_mnt+0x36/0x70
> [<0>] task_work_run+0x79/0xa0
> [<0>] exit_to_usermode_loop+0x62/0x70
> [<0>] do_syscall_64+0xdb/0xf0
> [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [<0>] 0xffffffffffffffff
Makes sense, please make sure to update the commit message.
I pushed wip-umount-force and gave it a quick test. The filesystem
part appears to need improvement -- lots of "VFS: Busy inodes after
unmount of ceph", more often than not with various crashes. See the
attached.
Thanks,
Ilya
[ 1239.752113] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.753045] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.753925] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.754828] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.755728] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.756633] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.757534] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.758481] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.759464] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.760857] ceph: writepage_start 000000004280d596 1099511627776 forced umount
[ 1239.816191] ceph: dropping dirty Fw state for 000000004280d596 1099511627776
[ 1239.817209] ceph: dropping dirty+flushing Fw state for 000000004280d596 1099511627776
[ 1239.819095] ceph: invalidate_pages 000000004280d596 1099511627776 forced umount
[ 1239.823689] VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds. Have a nice day...
[ 1239.995759] BUG: unable to handle kernel NULL pointer dereference at 0000000000000150
[ 1239.996454] PGD 0 P4D 0
[ 1239.996672] Oops: 0000 [#1] PREEMPT SMP PTI
[ 1239.997014] Modules linked in:
[ 1239.997268] CPU: 1 PID: 19 Comm: kworker/1:0 Not tainted 4.17.0-rc3-vm+ #112
[ 1239.997839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150318_183358- 04/01/2014
[ 1239.998612] Workqueue: ceph-pg-invalid ceph_invalidate_work
[ 1239.999067] RIP: 0010:ceph_check_caps+0x41/0xa00
[ 1239.999443] RSP: 0018:ffffc900000a3b00 EFLAGS: 00010286
[ 1239.999869] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
[ 1240.000444] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff880073ce0898
[ 1240.001020] RBP: ffff880073ce0d88 R08: 0000000000000000 R09: 0000000000000000
[ 1240.001593] R10: ffffc900000a3b50 R11: 47dbc355d3ae36f2 R12: ffff880073ce08a8
[ 1240.002168] R13: ffff880073ce0898 R14: 0000000000000000 R15: 0000000000000000
[ 1240.002744] FS: 0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[ 1240.003414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1240.003882] CR2: 0000000000000150 CR3: 0000000076418000 CR4: 00000000000006a0
[ 1240.004456] Call Trace:
[ 1240.004664] ? find_held_lock+0x2d/0x90
[ 1240.004979] ? ceph_put_wrbuffer_cap_refs+0x393/0x3c0
[ 1240.005386] ceph_put_wrbuffer_cap_refs+0x20f/0x3c0
[ 1240.005784] ceph_invalidatepage+0xcb/0xf0
[ 1240.006119] truncate_cleanup_page+0x64/0xa0
[ 1240.006468] truncate_inode_pages_range+0x18e/0x6c0
[ 1240.006866] truncate_pagecache+0x43/0x60
[ 1240.007193] ceph_invalidate_work+0x9d/0x200
[ 1240.007541] process_one_work+0x2ee/0x5b0
[ 1240.007870] worker_thread+0x20a/0x390
[ 1240.008176] ? process_one_work+0x5b0/0x5b0
[ 1240.008516] kthread+0x122/0x130
[ 1240.008784] ? kthread_create_worker_on_cpu+0x50/0x50
[ 1240.009193] ret_from_fork+0x3a/0x50
[ 1240.009485] Code: f3 48 81 ec 98 00 00 00 48 8b 87 18 05 00 00 48 8b 80 78 06 00 00 4c 8b 70 28 48 8d 87 f0 04 00 00 48 89 44 24 38 b8 01 00 00 00 <41> 8b ae 50 01 00 00 85 ed 0f 44 c6 83 e0 01 88 44 24 33 48 8d
[ 1240.011020] RIP: ceph_check_caps+0x41/0xa00 RSP: ffffc900000a3b00
[ 1240.011509] CR2: 0000000000000150
[ 1240.011784] ---[ end trace 39e8d9f1dfbee59c ]---
