On Sat, Mar 26, 2016 at 03:09:27PM -0400, Andrew Ryder wrote:
> Hello,
>
> I have an mdadm array with a xfs v5 filesystem on it and its begun to give
> me issues when trying to mount it as well as complete xfs_repair. Not sure
> if anyone might be able to shed some light on what is going on or how to
> correct the issue?
>
> When I try and mount the fs, it complains with:
>
> [ 388.479847] XFS (md2): Mounting V5 Filesystem
> [ 388.494686] XFS (md2): metadata I/O error: block 0x15d6d39c0
> ("xlog_bread_noalign") error 5 numblks 8192
> [ 388.495013] XFS (md2): failed to find log head
> [ 388.495018] XFS (md2): log mount/recovery failed: error -5
> [ 388.495090] XFS (md2): log mount failed
>
So a read I/O error from the kernel...
>
> This is where its not making any sense for me, If I try and run "xfs_repair
> /dev/md2" it fails with:
>
> Phase 1 - find and verify superblock...
> - reporting progress in intervals of 15 minutes
> Phase 2 - using internal log
> - zero log...
> xfs_repair: read failed: Input/output error
> failed to find log head
> zero_log: cannot find log head/tail (xlog_find_tail=-5)
>
> fatal error -- ERROR: The log head and/or tail cannot be discovered. Attempt
> to mount the
> filesystem to replay the log or use the -L option to destroy the log and
> attempt a repair.
>
... similar read error from xfsprogs...
>
> But if I run "xfs_repair -L /dev/md2" which gives:
>
> Phase 1 - find and verify superblock...
> - reporting progress in intervals of 15 minutes
> Phase 2 - using internal log
> - zero log...
> xfs_repair: read failed: Input/output error
> failed to find log head
> zero_log: cannot find log head/tail (xlog_find_tail=-5)
> xfs_repair: libxfs_device_zero write failed: Input/output error
>
... and it looks like it fails to write as well when trying to zero the
log...
> then try and re-run "xfs_repair /dev/md2" it starts traversing the
> filesystem all the way to "Phase 7" then errors with:
>
> Phase 7 - verify and correct link counts...
> - 14:36:55: verify and correct link counts - 33 of 33 allocation
> groups done
> Maximum metadata LSN (64:2230592) is ahead of log (0:0).
> Format log to cycle 67.
> xfs_repair: libxfs_device_zero write failed: Input/output error
>
>
> Yet at this point I can now mount the filesystem..
>
... and this is effectively a repeat of the write error as we try to
format the log with a correct LSN based on the metadata LSN tracked by
the repair process. Your kernel is old enough that runtime probably
won't complain either way (note that 3.19 might be considered a fairly
early kernel for using CRC support). Perhaps the first write attempt
zeroed enough of the log before it failed that log recovery wasn't
required, and thus these problematic I/Os were avoided.
What's the history of this fs? Has it been working for some time, just
recently formatted? What lead to the need for log recovery? What's the
mdadm --detail info, member device size, total array size, xfs_info of
the filesystem, etc..?
Does xfs_repair run clean at this point? If so, does 'xfs_repair -L'
still reproduce the write error (note that I'm assuming you have a clean
log such that this command will not cause data loss). If so, an strace
of the repair process might be interesting...
Brian
>
> Checking the drives with smartctl shows no errors nor does 'dmesg' show any
> hardware i/o or controller related errors...
>
> I've tried scrubbing the array and no bad sectors are found either..
>
> I'm running kernel 3.19.8 with xfsprogs 4.5.
>
> Thanks,
> Andrew
>
> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
