Hi Carlos,
Thanks for your reply!
Please see my responses in line below:
On 3/1/20, 3:08 am, "linux-xfs-owner@xxxxxxxxxxxxxxx on behalf of Carlos Maiolino" <linux-xfs-owner@xxxxxxxxxxxxxxx on behalf of cmaiolino@xxxxxxxxxx> wrote:
Hello.
> I’ve not got a lot of experience with XFS, so please be gentle.
>
> I’ve got 2 external HDD’s mounted through ESXi to a machine called USB-3 which each have vdo running on them and then a Logical Volume called vdovg-vdolvm on it.
I believe you're talking about VMWare ESXi and you are mounting these external
HDDs into a virtual machine.
Yes - this is correct.
>
> It’s currently unable to be mounted. It gives the error:
>
> [daniel.storey@usb-3 ~]$ sudo mount /data
> mount: mount /dev/mapper/vdovg-vdolvm on /data failed: Structure needs cleaning
> And then when I try to run xfs_repair it throws the following error:
Ok, so, sounds like your FS is corrupted?!
Yep - I think it is.
>
> [daniel.storey@usb-3 ~]$ sudo xfs_repair /dev/dm-4
> Phase 1 - find and verify superblock...
> superblock read failed, offset 6597069742080, size 131072, ag 6, rval -1
>
> fatal error -- Input/output error
You are having I/O errors on your device. Have you ever tried to mount this
device on a machine other than VMWare hosted machine? Bare-metal specially.
No I haven't, but I've run smartctl -t long /dev/sdb and sdc and they're not showing any errors.
Also, why are you pointing /dev/dm-4 directly, instead of using the VDO link you
mentioned above?
No reason - if I run the command with the vdo link it shows the same error.
>
> However, I’m able to view the file system with ufs explorer, so I think it’s still there.
>
Also on a vmware machine? On the same hypervisor? For sure not on the same host,
since UFS explorer (AFAIK) does not have a Linux version.
It does, actually (have a Linux version). I'm running it on the same host.
And btw, UFS Explorer is built so that you can scan/recover data on very damaged
filesystems and disks, while filesystems won't let you mount a corrupted
filesystem to avoid doing even more damage. So, yeah, you might still see
filesystem data/metadata using UFS explorer with damaged filesystems or block
devices.
Okay.
Now, looking at the dmesg output you sent:
> [52244.526969] kvdo1:logQ0: Completing read VIO for LBN 1610612991 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52244.526978] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52244.527440] kvdo1:logQ0: Completing read VIO for LBN 1610612990 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52244.527447] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52244.527851] kvdo1:logQ0: Completing read VIO for LBN 1610612987 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52244.527856] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52316.831349] kvdo1:logQ0: Completing read VIO for LBN 1610612991 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52316.831364] kvdo1:cpuQ1: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52316.832085] kvdo1:logQ0: Completing read VIO for LBN 1610612990 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52316.832092] kvdo1:cpuQ1: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52316.832802] kvdo1:logQ0: Completing read VIO for LBN 1610612987 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52316.832809] kvdo1:cpuQ1: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52819.634153] kvdo1:logQ0: Completing read VIO for LBN 1610612987 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52819.634177] kvdo1:cpuQ1: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [52819.637152] kvdo1:logQ0: Completing read VIO for LBN 1610612987 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [52819.637173] kvdo1:cpuQ1: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
Even VDO driver is having problems to issue IO to your device. So either your
device is reaching End-Of-Life, or VMWare has something to do there which is
causing IO Errors.
So, again, I'd try to open these devices on a bare-metal machine and check the
device for errors. If the errors are still present, replace the devices.
Ok - I'll try opening these devices on a bare-metal (not a VMware host) and check them for errors. What do I do if there are no errors present? As the SMART check revealed no problems with the disks.
Thanks!
Cheers.
> [52819.637179] Buffer I/O error on dev dm-4, logical block 1610612731, async page read
> [54235.537518] XFS (dm-4): Mounting V5 Filesystem
> [54236.686544] XFS (dm-4): Starting recovery (logdev: internal)
> [54236.966501] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.968126] XFS (dm-4): Unmount and run xfs_repair
> [54236.968683] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.969267] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.969836] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.970422] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.971012] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.971582] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.972721] XFS (dm-4): Unmount and run xfs_repair
> [54236.973297] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.973843] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.974433] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.975009] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.975564] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.976161] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.977296] XFS (dm-4): Unmount and run xfs_repair
> [54236.977837] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.978414] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.978993] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.979547] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.980129] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.980701] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.981835] XFS (dm-4): Unmount and run xfs_repair
> [54236.982407] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.982952] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.983544] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.984119] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.984671] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.985268] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.986400] XFS (dm-4): Unmount and run xfs_repair
> [54236.986943] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.987517] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.988094] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.988620] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.989178] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.989719] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.990808] XFS (dm-4): Unmount and run xfs_repair
> [54236.991373] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.991911] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.992483] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.993041] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.993568] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.994139] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.995228] XFS (dm-4): Unmount and run xfs_repair
> [54236.995761] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54236.996329] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54236.996870] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54236.997428] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54236.997954] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54236.998529] XFS (dm-4): Metadata corruption detected at xfs_inode_buf_verify+0x79/0x100 [xfs], xfs_inode block 0x30064db50
> [54236.999621] XFS (dm-4): Unmount and run xfs_repair
> [54237.000183] XFS (dm-4): First 64 bytes of corrupted metadata buffer:
> [54237.000727] ffffb9ad71884000: 49 4e 81 a4 03 02 00 00 00 00 00 00 00 00 00 00 IN..............
> [54237.001469] ffffb9ad71884010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [54237.002026] ffffb9ad71884020: 5d e1 4f 12 1b a2 b7 8a 5d e1 4f 11 37 5a 0d c5 ].O.....].O.7Z..
> [54237.002553] ffffb9ad71884030: 5d e1 4f 11 37 5a 0d c5 00 00 00 00 01 50 50 00 ].O.7Z.......PP.
> [54237.003195] XFS (dm-4): metadata I/O error: block 0x30064db50 ("xlog_recover_do..(read#2)") error 117 numblks 32
> [54237.004631] XFS (dm-4): log mount/recovery failed: error -117
> [54237.004945] XFS (dm-4): log mount failed
> [54289.229381] kvdo1:logQ0: Completing read VIO for LBN 1610612991 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [54289.229422] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [54289.230502] kvdo1:logQ0: Completing read VIO for LBN 1610612990 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [54289.230527] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
> [54289.231126] kvdo1:logQ0: Completing read VIO for LBN 1610612987 with error after readData: kvdo: Compressed block fragment is invalid (2073)
> [54289.231147] kvdo1:cpuQ0: mapToSystemError: mapping internal status code 2073 (kvdo: VDO_INVALID_FRAGMENT: kvdo: Compressed block fragment is invalid) to EIO
--
Carlos
