Dear Sirs
I wonder if you can help with an issue we see re-occuring on a regular basis with one of our HP systems which uses a HP 420 Raid controller
Action taken
We first saw the following:
[root@ content]# ls
ls: cannot open directory .: Input/output error
[root@ /]# ls -ltr
ls: cannot access content
total 358
d?????????? ? ? ? ? ? content
drwxr-xr-x. 2 root root 4096 Jun 28 2011 srv
drwxr-xr-x. 2 root root 4096 Jun 28 2011 media
drwxr-xr-x. 2 root root 4096 Feb 22 2012 cgroup
drwx------. 2 root root 16384 Jul 21 2012 lost+found
drwxr-xr-x. 2 root root 4096 Jul 21 2012 selinux
We try to run:
[root@ /]# xfs_check /dev/md0
xfs_check: /dev/md0 contains a mounted and writable filesystem
fatal error -- couldn't initialize XFS library
We also tried to umount the /dev/md0 before runniing xfs_check but no luck. We received the error: device is in use
We use xfs for one of our large raid file systems and we are seeing the xfs filesystem go offline with the following messages in dmesg
messages-20140921:Sep 18 23:01: kernel: XFS (md0): Device md0: metadata write error block 0x5e28623d8
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): I/O error occurred: meta-data dev md0 block 0x445cccc40 ("xlog_iodone") error 5 buf count 32768
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa2c428dc
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): Log I/O Error Detected. Shutting down filesystem
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): Please umount the filesystem and rectify the problem(s)
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_iunlink_remove: xfs_itobp() returned error 5.
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): I/O error occurred: meta-data dev md0 block 0x445cccc80 ("xlog_iodone") error 5 buf count 32768
messages-20140921:Sep 18 23:01:04 kernel: XFS (md0): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa2c428dc
XFS (md0): xfs_log_force: error 5 returned.
XFS (md0): xfs_log_force: error 5 returned.
XFS (md0): xfs_log_force: error 5 returned.
In all occurrences the only way to recover from this is to reboot the system and allow xfs_repair to run during boot this clears the issue until next time
We have checked the RAID health and nothing seems to be amiss, if you could help with this it would be much appreciated
Best regards Simon
Simon Dray
s
p: +44.1223 716.400
p: +44.1223 716.476
e: sdray@xxxxxxxxxx
1st Floor, 335 Cambridge Science Park, Milton Road, Cambridge, Cambridgeshire, CB4 0WN, United Kingdom.
Understanding is a three-edged sword
