Hi! Yesterday I got the following error messages from the kernel (Ubuntu trusty, 3.13.0-55): [601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................ [601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... .......... [601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [601201.820013] XFS (dm-0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00996f0 [601201.820910] CPU: 1 PID: 421 Comm: xfsaild/dm-0 Not tainted 3.13.0-55-generic #92-Ubuntu [601201.820913] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011 [601201.820914] 0000000000000001 ffff880035cc3bd0 ffffffff81723294 ffff8802ac71d800 [601201.820918] ffff880035cc3be8 ffffffffa009d6fb ffffffffa00996f0 ffff880035cc3c20 [601201.820919] ffffffffa009d755 000000d800205500 ffff88001aa2a400 ffff880050acc2b8 [601201.820922] Call Trace: [601201.820930] [<ffffffff81723294>] dump_stack+0x45/0x56 [601201.820964] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs] [601201.820974] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs] [601201.820984] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs] [601201.820997] [<ffffffffa00bab50>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs] [601201.821007] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs] [601201.821016] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs] [601201.821026] [<ffffffffa00996f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs] [601201.821030] [<ffffffff8109abc0>] ? wake_up_state+0x20/0x20 [601201.821040] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs] [601201.821050] [<ffffffffa009b336>] xfs_buf_iorequest+0x46/0x90 [xfs] [601201.821060] [<ffffffffa009b3d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs] [601201.821070] [<ffffffffa009b56b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs] [601201.821081] [<ffffffffa009c000>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [601201.821100] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [601201.821110] [<ffffffffa009c000>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs] [601201.821127] [<ffffffffa00facd7>] xfsaild+0x237/0x5c0 [xfs] [601201.821145] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [601201.821148] [<ffffffff8108b702>] kthread+0xd2/0xf0 [601201.821150] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [601201.821153] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90 [601201.821155] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [601201.821158] XFS (dm-0): Corruption detected. Unmount and run xfs_repair [601201.821581] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa009971c [601201.832780] XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem [601201.833292] XFS (dm-0): Please umount the filesystem and rectify the problem(s) [601201.843375] ffff880238b00000: 58 44 32 42 01 00 0e c0 00 00 00 00 00 00 00 00 XD2B............ [601201.843924] ffff880238b00010: 00 00 00 20 3d a7 0c 7b 01 2e 8d 9e 01 88 00 10 ... =..{........ [601201.844472] ffff880238b00020: 00 00 00 19 21 a9 2d 2f 02 2e 2e 9e 01 88 00 20 ....!.-/....... [601201.845031] ffff880238b00030: 00 00 00 07 39 2b 68 20 18 4d 4f 42 31 41 5f 48 ....9+h .MOB1A_H [601201.845585] XFS (dm-0): Internal error xfs_da3_node_read_verify at line 240 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_da_btree.c. Caller 0xffffffffa009a885 [601201.858885] CPU: 1 PID: 2806 Comm: kworker/1:1H Not tainted 3.13.0-55-generic #92-Ubuntu [601201.858889] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011 [601201.858938] Workqueue: xfslogd xfs_buf_iodone_work [xfs] [601201.858941] 0000000000000001 ffff8802af9cbd68 ffffffff81723294 ffff8802ac71d800 [601201.858944] ffff8802af9cbd80 ffffffffa009d6fb ffffffffa009a885 ffff8802af9cbdb8 [601201.858945] ffffffffa009d755 000000f0810a0465 ffff880031b35b00 ffff880238b00000 [601201.858948] Call Trace: [601201.858958] [<ffffffff81723294>] dump_stack+0x45/0x56 [601201.858969] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs] [601201.858979] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [601201.858989] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs] [601201.859005] [<ffffffffa00d146c>] xfs_da3_node_read_verify+0x8c/0x180 [xfs] [601201.859014] [<ffffffffa009a885>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs] [601201.859024] [<ffffffffa009a885>] xfs_buf_iodone_work+0x85/0xf0 [xfs] [601201.859028] [<ffffffff81083b22>] process_one_work+0x182/0x450 [601201.859030] [<ffffffff81084911>] worker_thread+0x121/0x410 [601201.859032] [<ffffffff810847f0>] ? rescuer_thread+0x430/0x430 [601201.859035] [<ffffffff8108b702>] kthread+0xd2/0xf0 [601201.859037] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [601201.859040] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90 [601201.859042] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0 [601201.859044] XFS (dm-0): Corruption detected. Unmount and run xfs_repair [601201.872288] XFS (dm-0): metadata I/O error: block 0x101ed393c0 ("xfs_trans_read_buf_map") error 117 numblks 8 [601201.942054] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5. [601209.420268] XFS (dm-0): xfs_log_force: error 5 returned. [601239.529710] XFS (dm-0): xfs_log_force: error 5 returned. [601269.639150] XFS (dm-0): xfs_log_force: error 5 returned. [601299.748593] XFS (dm-0): xfs_log_force: error 5 returned. [601329.858034] XFS (dm-0): xfs_log_force: error 5 returned. I'm currently running xfs_repair -n and so far I've seen the following output: Phase 1 - find and verify superblock... Phase 2 - using internal log - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 doubling cache size to 591200 directory flags set on non-directory inode 206624 directory flags set on non-directory inode 206625 directory flags set on non-directory inode 206626 directory flags set on non-directory inode 206627 directory flags set on non-directory inode 206628 directory flags set on non-directory inode 206629 directory flags set on non-directory inode 206630 directory flags set on non-directory inode 206631 directory flags set on non-directory inode 206632 directory flags set on non-directory inode 206633 directory flags set on non-directory inode 206634 Only the first 20 lines are included. There are currently 250000+ more lines with "directory flags set on non-directory inode" and the check is still running (the mostly small files take up around 30 TB, so it'll probably take a while). I recently enabled user and project quota and updated from 3.13.0-53. The file system has been heavily used for the last month or so. Does anyone have any thoughts on this? I'm tempted to stop using quotas when the file system (hopefully) works again, as it's my impression that project quotas are not widely used. Best, Rasmus
Intomics is a contract research organization specialized in deriving core biological insight from large scale data. We help our clients in the pharmaceutical industry develop tomorrow's medicines better, faster, and cheaper through optimized use of biomedical data.
----------------------------------------------------------------- Hansen, Rasmus Borup Intomics - from data to biology System Administrator Diplomvej 377 Scientific Programmer DK-2800 Kgs. Lyngby Denmark P: +45 5167 7972 P: +45 8880 7979 |
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
