Dear xfs experts, We're running glusterfs over top of xfs and recently we have encountered xfs filesystem crash two times on two different servers. Can you kindly help to give me some insight of how to debug this kind of failure? Bellow is xfs filesystem crash messages: server1: Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_iunlink_remove: xfs_inotobp() returned error 22. Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_inactive: xfs_ifree returned error 22 Dec 28 16:55:01 localhost kernel: XFS (dm-0): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa029f20a Dec 28 16:55:01 localhost kernel: XFS (dm-0): I/O Error Detected. Shutting down filesystem Dec 28 16:55:01 localhost kernel: XFS (dm-0): Please umount the filesystem and rectify the problem(s) Dec 28 16:55:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:55:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:56:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:56:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:57:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:57:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:58:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:58:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:59:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 16:59:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:00:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:00:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:01:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:01:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:02:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:02:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:03:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:03:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:04:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:04:20 localhost sshd[20609]: Accepted publickey for fuyongtao from 10.1.12.99 port 56514 ssh2 Dec 28 17:04:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:05:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:05:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:06:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:06:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:07:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:07:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:08:20 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:08:50 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_log_force: error 5 returned. Dec 28 17:08:55 localhost kernel: XFS (dm-0): xfs_do_force_shutdown(0x1) called from line 1056 of file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa02a6213 server2: Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_iunlink_remove: xfs_inotobp() returned error 22. Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_inactive: xfs_ifree returned error 22 Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_do_force_shutdown(0x1) called from line 1184 of file fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa022420a Dec 27 00:30:02 localhost kernel: XFS (sdb): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5. Dec 27 00:30:02 localhost kernel: XFS (sdb): I/O Error Detected. Shutting down filesystem Dec 27 00:30:02 localhost kernel: XFS (sdb): Please umount the filesystem and rectify the problem(s) Dec 27 00:30:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:30:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:31:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:31:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:32:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:32:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:33:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:33:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:34:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:34:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:35:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:35:32 localhost smbd[25481]: [2012/12/27 00:35:32.424643, 0] printing/print_cups.c:109(cups_connect) Dec 27 00:35:32 localhost smbd[25481]: Unable to connect to CUPS server localhost:631 - Connection refused Dec 27 00:35:32 localhost smbd[2061]: [2012/12/27 00:35:32.424906, 0] printing/print_cups.c:468(cups_async_callback) Dec 27 00:35:32 localhost smbd[2061]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL Dec 27 00:35:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:36:27 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. Dec 27 00:36:57 localhost kernel: XFS (sdb): xfs_log_force: error 5 returned. -- 符永涛 _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
