On Fri, Jun 17, 2016 at 5:18 AM, Adam Tygart <mozes@xxxxxxx> wrote: > This sounds an awful lot like a a bug I've run into a few times (not > often enough to get a good backtrace out of the kernel or mds) > involving vim on a symlink to a file in another directory. It will > occasionally corrupt the symlink in such a way that the symlink is > unreadable. Filling dmesg with: > > [ 2368.036667] ceph: fill_inode badness on ffff8800bb5fb610 > [ 2368.969657] ------------[ cut here ]------------ > [ 2368.969670] WARNING: CPU: 0 PID: 15 at fs/ceph/inode.c:813 > fill_inode.isra.19+0x4b1/0xa49() > [ 2368.969672] Modules linked in: > [ 2368.969684] CPU: 0 PID: 15 Comm: kworker/0:1 Tainted: G W > 4.5.0-gentoo #1 > [ 2368.969686] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > [ 2368.969693] Workqueue: ceph-msgr ceph_con_workfn > [ 2368.969695] 0000000000000286 000000007000a7b9 ffff88017e267af0 > ffffffffb142ec39 > [ 2368.969698] 0000000000000000 0000000000000009 ffff88017e267b28 > ffffffffb1091c83 > [ 2368.969700] ffffffffb13be512 ffffc900020da8cd ffff880427a30230 > ffffffffffffffff > [ 2368.969704] Call Trace: > [ 2368.969709] [<ffffffffb142ec39>] dump_stack+0x63/0x7f > [ 2368.969714] [<ffffffffb1091c83>] warn_slowpath_common+0x9a/0xb3 > [ 2368.969717] [<ffffffffb13be512>] ? fill_inode.isra.19+0x4b1/0xa49 > [ 2368.969719] [<ffffffffb1091d86>] warn_slowpath_null+0x15/0x17 > [ 2368.969722] [<ffffffffb13be512>] fill_inode.isra.19+0x4b1/0xa49 > [ 2368.969724] [<ffffffffb13bca00>] ? ceph_mount+0x729/0x72e > [ 2368.969727] [<ffffffffb13bf705>] ceph_readdir_prepopulate+0x48f/0x70c > [ 2368.969730] [<ffffffffb13daac3>] dispatch+0xebf/0x1428 > [ 2368.969752] [<ffffffffb19098f2>] ? ceph_x_check_message_signature+0x42/0xc4 > [ 2368.969756] [<ffffffffb18fa16e>] ceph_con_workfn+0xe1a/0x24f3 > [ 2368.969759] [<ffffffffb104603a>] ? load_TLS+0xb/0xf > [ 2368.969761] [<ffffffffb10468f9>] ? __switch_to+0x3b0/0x42b > [ 2368.969765] [<ffffffffb10afd8f>] ? finish_task_switch+0xff/0x191 > [ 2368.969768] [<ffffffffb10a53b3>] process_one_work+0x175/0x2a0 > [ 2368.969770] [<ffffffffb10a59c8>] worker_thread+0x1fc/0x2ae > [ 2368.969772] [<ffffffffb10a57cc>] ? rescuer_thread+0x2c0/0x2c0 > [ 2368.969775] [<ffffffffb10a9c4b>] kthread+0xaf/0xb7 > [ 2368.969777] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2368.969780] [<ffffffffb192620f>] ret_from_fork+0x3f/0x70 > [ 2368.969782] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2368.969784] ---[ end trace b054c5c6854fd2ab ]--- > [ 2368.969786] ceph: fill_inode badness on ffff880428185d70 > [ 2370.289733] ------------[ cut here ]------------ > [ 2370.289747] WARNING: CPU: 0 PID: 15 at fs/ceph/inode.c:813 > fill_inode.isra.19+0x4b1/0xa49() > [ 2370.289750] Modules linked in: > [ 2370.289756] CPU: 0 PID: 15 Comm: kworker/0:1 Tainted: G W > 4.5.0-gentoo #1 > [ 2370.289759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > [ 2370.289767] Workqueue: ceph-msgr ceph_con_workfn > [ 2370.289769] 0000000000000286 000000007000a7b9 ffff88017e267af0 > ffffffffb142ec39 > [ 2370.289774] 0000000000000000 0000000000000009 ffff88017e267b28 > ffffffffb1091c83 > [ 2370.289777] ffffffffb13be512 ffffc900020f58cd ffff880427a30230 > ffffffffffffffff > [ 2370.289781] Call Trace: > [ 2370.289787] [<ffffffffb142ec39>] dump_stack+0x63/0x7f > [ 2370.289793] [<ffffffffb1091c83>] warn_slowpath_common+0x9a/0xb3 > [ 2370.289797] [<ffffffffb13be512>] ? fill_inode.isra.19+0x4b1/0xa49 > [ 2370.289801] [<ffffffffb1091d86>] warn_slowpath_null+0x15/0x17 > [ 2370.289804] [<ffffffffb13be512>] fill_inode.isra.19+0x4b1/0xa49 > [ 2370.289807] [<ffffffffb13bca00>] ? ceph_mount+0x729/0x72e > [ 2370.289811] [<ffffffffb13bf705>] ceph_readdir_prepopulate+0x48f/0x70c > [ 2370.289815] [<ffffffffb13daac3>] dispatch+0xebf/0x1428 > [ 2370.289821] [<ffffffffb19098f2>] ? ceph_x_check_message_signature+0x42/0xc4 > [ 2370.289824] [<ffffffffb18fa16e>] ceph_con_workfn+0xe1a/0x24f3 > [ 2370.289829] [<ffffffffb104603a>] ? load_TLS+0xb/0xf > [ 2370.289832] [<ffffffffb10468f9>] ? __switch_to+0x3b0/0x42b > [ 2370.289837] [<ffffffffb10afd8f>] ? finish_task_switch+0xff/0x191 > [ 2370.289841] [<ffffffffb10a53b3>] process_one_work+0x175/0x2a0 > [ 2370.289843] [<ffffffffb10a59c8>] worker_thread+0x1fc/0x2ae > [ 2370.289846] [<ffffffffb10a57cc>] ? rescuer_thread+0x2c0/0x2c0 > [ 2370.289849] [<ffffffffb10a9c4b>] kthread+0xaf/0xb7 > [ 2370.289853] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2370.289857] [<ffffffffb192620f>] ret_from_fork+0x3f/0x70 > [ 2370.289860] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2370.289863] ---[ end trace b054c5c6854fd2ac ]--- > [ 2370.289865] ceph: fill_inode badness on ffff880428185d70 > [ 2371.525649] ------------[ cut here ]------------ > [ 2371.525663] WARNING: CPU: 0 PID: 15 at fs/ceph/inode.c:813 > fill_inode.isra.19+0x4b1/0xa49() > [ 2371.525665] Modules linked in: > [ 2371.525670] CPU: 0 PID: 15 Comm: kworker/0:1 Tainted: G W > 4.5.0-gentoo #1 > [ 2371.525672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > [ 2371.525679] Workqueue: ceph-msgr ceph_con_workfn > [ 2371.525682] 0000000000000286 000000007000a7b9 ffff88017e267af0 > ffffffffb142ec39 > [ 2371.525685] 0000000000000000 0000000000000009 ffff88017e267b28 > ffffffffb1091c83 > [ 2371.525687] ffffffffb13be512 ffffc900021108cd ffff880427a30230 > ffffffffffffffff > [ 2371.525690] Call Trace: > [ 2371.525696] [<ffffffffb142ec39>] dump_stack+0x63/0x7f > [ 2371.525701] [<ffffffffb1091c83>] warn_slowpath_common+0x9a/0xb3 > [ 2371.525704] [<ffffffffb13be512>] ? fill_inode.isra.19+0x4b1/0xa49 > [ 2371.525707] [<ffffffffb1091d86>] warn_slowpath_null+0x15/0x17 > [ 2371.525740] [<ffffffffb13be512>] fill_inode.isra.19+0x4b1/0xa49 > [ 2371.525744] [<ffffffffb13bca00>] ? ceph_mount+0x729/0x72e > [ 2371.525747] [<ffffffffb13bf705>] ceph_readdir_prepopulate+0x48f/0x70c > [ 2371.525751] [<ffffffffb13daac3>] dispatch+0xebf/0x1428 > [ 2371.525755] [<ffffffffb19098f2>] ? ceph_x_check_message_signature+0x42/0xc4 > [ 2371.525758] [<ffffffffb18fa16e>] ceph_con_workfn+0xe1a/0x24f3 > [ 2371.525762] [<ffffffffb104603a>] ? load_TLS+0xb/0xf > [ 2371.525764] [<ffffffffb10468f9>] ? __switch_to+0x3b0/0x42b > [ 2371.525769] [<ffffffffb10afd8f>] ? finish_task_switch+0xff/0x191 > [ 2371.525772] [<ffffffffb10a53b3>] process_one_work+0x175/0x2a0 > [ 2371.525774] [<ffffffffb10a59c8>] worker_thread+0x1fc/0x2ae > [ 2371.525776] [<ffffffffb10a57cc>] ? rescuer_thread+0x2c0/0x2c0 > [ 2371.525779] [<ffffffffb10a9c4b>] kthread+0xaf/0xb7 > [ 2371.525782] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2371.525786] [<ffffffffb192620f>] ret_from_fork+0x3f/0x70 > [ 2371.525788] [<ffffffffb10a9b9c>] ? kthread_parkme+0x1f/0x1f > [ 2371.525790] ---[ end trace b054c5c6854fd2ad ]--- > see http://tracker.ceph.com/issues/16013. It was caused by bad symlink size. Creating the symlink can resolve this issue. Regards Yan, Zheng > Whenever a readdir is performed on the directory containing the > symlink, and all the stats go ??????? and the symlink is unable to be > deleted/moved/operated on. > > I believe it involves the overwrites that vim performs on save (save > to temporary file and move it overtop of existing, I believe). I've > seen it on kernels 4.0->4.5 so far. Possibly even earlier. > Hammer->Infernalis, I've not had a chance to test on Jewel. > > I'd dump the symlink data out of the metadata pool, but I'm still > recovering from http://tracker.ceph.com/issues/16177 > > Not trying to hijack your thread here, though. > > -- > Adam > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |