On 4/9/13 7:53 AM, 符永涛 wrote: > Dear xfs experts, > I really need your help sincerely!!! In our production enviroment we > run glusterfs over top of xfs on Dell x720D(Raid 6). And the xfs file > system crash on some of the server frequently about every two weeks. > Can you help to give me a direction about how to debug this issue and > how to avoid it? Thank you very very much! So this happens reliably, but infrequently? (only every 2 weeks or so?) Can you provoke it any more often? > uname -a > Linux cqdx.miaoyan.cluster1.node11.qiyi.domain 2.6.32-279.el6.x86_64 > #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux That's a RHEL6 kernel; I'm assuming that this is a RHEL clone w/o RH support? I agree with Ben that I'd like to see xfs_repair output. Since the fs has shut down, you should unmount, remount, and unmount again to replay the dirty log. Then do xfs_repair -n, and provide the output if it discovers any errors. Thanks, -Eric > Every time the crash log is same, as following > > 038 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_iunlink_remove: > xfs_inotobp() returned error 22. > 1039 Apr 9 09:41:36 cqdx kernel: XFS (sdb): xfs_inactive: xfs_ifree > returned error 22 > 1040 Apr 9 09:41:36 cqdx kernel: XFS (sdb): > xfs_do_force_shutdown(0x1) called from line 1184 of file > fs/xfs/xfs_vnodeops.c. Return address = 0xffffffffa02ee20a > 1041 Apr 9 09:41:36 cqdx kernel: XFS (sdb): I/O Error Detected. > Shutting down filesystem > 1042 Apr 9 09:41:36 cqdx kernel: XFS (sdb): Please umount the > filesystem and rectify the problem(s) > 1043 Apr 9 09:41:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned. > 1044 Apr 9 09:42:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned. > 1045 Apr 9 09:42:53 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned. > 1046 Apr 9 09:43:23 cqdx kernel: XFS (sdb): xfs_log_force: error 5 returned. > _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs