On Wed, Mar 8, 2017 at 8:01 PM, Olivier Swinkels
<olivier.swinkels@xxxxxxxxx> wrote:
> On Tue, Mar 7, 2017 at 3:52 PM, Phil Turmel <philip@xxxxxxxxxx> wrote:
>> On 03/07/2017 03:39 AM, Olivier Swinkels wrote:
>>
>>> After I used the pvcreate command to recreate pv the vgcfgrestore
>>> command succeeds and the lvm is available (after activating).
>>>
>>> However when I try to mount it I get the following error: sudo mount
>>> -t ext4 /dev/lvm-raid/lvm0 /mnt/raid mount: mount
>>> /dev/mapper/lvm--raid-lvm0 on /mnt/raid failed: Structure needs
>>> cleaning
>>>
>>> So I guess the underlying RAID array is still not ok...
>>
>> No, your underlying array is very likely correct. But the intervening
>> incorrect --create operation stomped on your filesystems. Run fsck
>> while unmounted to deal with the corruption and recover what you can.
>>
>> Run fsck with "-n" first, to see just how extensive the problems are,
>> then with "-y" to actually fix things. Based on your sequence of
>> events, your corruptions should be at low sector addresses (first few
>> Gigs) of your array. If that's what appears with "-n", proceed.
>>
>> If you are unlucky, the stompage hit one or more of your filesystems'
>> superblocks, requiring access to backup superblocks. If you still see
>> no progress with either of the above, you might need to search your
>> array for ext2/3/4 superblocks. This grep would help:
>>
>> dd if=/dev/md0 bs=1M count=16k 2>/dev/null |hexdump -C |grep '30 .\+ 53 ef 0'
>>
>> (Not all hits from the grep will be superblocks, but they would be
>> visually distinguishable, and would have decipherable timestamps.)
>>
>> Phil
>
> Hi,
>
> I ran fsck -n disk on the lvm and got the very large response below.
> This didn't look promising, so I ran the ext2/3/4 superblock search.
> I didn't recognize any obvious timestamps, but i'm not sure what to look for.
> (I also pasted this output below)
>
> Can you recommend any further action?
>
> Olivier
>
> ===============================================================================
> fsck /dev/lvm-raid/lvm0
> fsck from util-linux 2.27.1
> ext2fs_check_desc: Corrupt group descriptor: bad block for block bitmap
> fsck.ext4: Group descriptors look bad... trying backup blocks...
> Warning: skipping journal recovery because doing a read-only filesystem check.
> /dev/mapper/lvm--raid-lvm0 has gone 1275 days without being checked,
> check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> Free blocks count wrong for group #0 (23513, counted=5372).
> Fix? no
> Free blocks count wrong for group #1 (980, counted=485).
> Fix? no
> Free blocks count wrong for group #3 (1023, counted=151).
> Fix? no
> Free blocks count wrong for group #5 (1022, counted=142).
> Fix? no
>
> <<TRIMMED ~200000 lines >>>
>
> Free inodes count wrong for group #60160 (8192, counted=8163).
> Fix? no
> Directories count wrong for group #60160 (0, counted=1).
> Fix? no
> Free inodes count wrong for group #60161 (8192, counted=8191).
> Fix? no
> Directories count wrong for group #60161 (0, counted=1).
> Fix? no
> Free inodes count wrong (488172170, counted=609035768).
> Fix? no
>
> /dev/mapper/lvm--raid-lvm0: ********** WARNING: Filesystem still has
> errors **********
> /dev/mapper/lvm--raid-lvm0: 122295670/610467840 files (0.0%
> non-contiguous), 2291846165/2441871360 blocks
> ===============================================================================
> dd if=/dev/md0 bs=1M count=16k 2>/dev/null |hexdump -C |grep '30 .\+ 53 ef 0'
> 00040430 3a 3b ab 58 17 00 19 00 53 ef 01 00 01 00 00 00 |:;.X....S.......|
> 08040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 18040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 1d220e30 2e 93 9c 90 c1 89 27 62 53 ef 03 12 d4 82 36 76 |......'bS.....6v|
> 27340430 bb 78 c4 e3 9e 56 62 0f 53 ef 04 b9 38 0c d0 26 |.x...Vb.S...8..&|
> 28040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 2f02ab30 5e d0 c9 98 83 ce 3b 92 53 ef 08 51 c4 4b dc af |^.....;.S..Q.K..|
> 38040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 44318c30 55 c1 19 f3 10 fb ab 2f 53 ef 04 0d fd c1 dc ed |U....../S.......|
> 48040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 50791730 6b 04 06 e8 97 88 9b 08 53 ef 0b 24 21 68 3d a5 |k.......S..$!h=.|
> 5aab3030 db 52 8d 5c 82 f4 80 cd 53 ef 08 4c f7 a7 c7 a9 |.R.\....S..L....|
> 98b69330 41 07 bb 64 c2 4a 00 5a 53 ef 08 71 88 d2 5a 42 |A..d.J.ZS..q..ZB|
> c8040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> cefc4130 14 37 b9 ef 1f 89 14 ab 53 ef 02 1c ca 88 f0 c4 |.7......S.......|
> d8040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> fe124d30 8a 6f 46 db 63 9b c5 9b 53 ef 07 97 74 24 2d e0 |.oF.c...S...t$-.|
> 106432f30 76 d0 fc 66 02 06 e9 50 53 ef 02 44 6c ab 89 ac |v..f...PS..Dl...|
> 111e92630 f8 59 df b2 af 5d 6b 8b 53 ef 0d 65 c6 29 66 0c |.Y...]k.S..e.)f.|
> 12883be30 88 63 b9 3a 6a 52 03 a6 53 ef 0c 12 67 dc ad 9e |.c.:jR..S...g...|
> 14d315530 30 92 ef 27 a0 dc 46 dd 53 ef 09 d7 89 5a ba 95 |0..'..F.S....Z..|
> 17222e430 aa 7d 4f 4a c5 88 fa f0 53 ef 0c 08 04 10 9e ad |.}OJ....S.......|
> 173717c30 3d fd 91 37 5a 0e 7b aa 53 ef 0b 08 d4 51 89 fa |=..7Z.{.S....Q..|
> 17665ae30 c9 a8 24 d6 ba 0a 50 f7 53 ef 08 13 da d7 52 0a |..$...P.S.....R.|
> 188040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 18ef3be30 41 c3 91 02 65 af 8d 27 53 ef 0c 1f 4b 47 f5 97 |A...e..'S...KG..|
> 1943f1d30 d6 87 e0 d7 c1 3b e7 53 53 ef 08 a5 88 14 b6 c8 |.....;.SS.......|
> 19bfba930 c3 e4 07 04 b4 d1 05 4a 53 ef 02 c4 69 2f e2 d0 |.......JS...i/..|
> 1a807c530 a1 12 db b0 77 9d c5 6c 53 ef 0d 6e 37 05 74 61 |....w..lS..n7.ta|
> 1b1976330 1a 2b 57 56 46 96 c2 9b 53 ef 03 78 0d 9e 4a e0 |.+WVF...S..x..J.|
> 1ebf5db30 1e 76 bf 9d 25 8f fd 16 53 ef 04 ba 23 14 2f 8d |.v..%...S...#./.|
> 1eec8b930 d5 dc 1e 90 b1 c8 f4 31 53 ef 06 10 3c 81 fb 37 |.......1S...<..7|
> 1f1392930 89 db 6c 92 6c 10 a5 f9 53 ef 05 31 0e 04 a2 d0 |..l.l...S..1....|
> 227cdff30 8d 8a 0a 61 91 04 c4 15 53 ef 03 5b 28 c4 52 59 |...a....S..[(.RY|
> 23000da30 1d e6 34 ca 2f 36 08 78 53 ef 0e 28 e4 94 5f 42 |..4./6.xS..(.._B|
> 245749e30 83 67 b7 d0 8b 40 0d f0 53 ef 0e ec 13 37 39 89 |.g...@..S....79.|
> 24f40e530 86 3b fc 43 ea ef 81 29 53 ef 09 6c fb 08 3b 6d |.;.C...)S..l..;m|
> 25a473430 94 24 8d 8c 6b 15 7b 8b 53 ef 00 ae 43 b2 64 a4 |.$..k.{.S...C.d.|
> 288040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
> 29966b530 00 ad 14 71 ea c5 c6 7a 53 ef 0c 11 d2 57 fb a0 |...q...zS....W..|
> 2a5924830 98 43 57 9b 2f 98 18 4f 53 ef 0d 27 72 c3 58 ba |.CW./..OS..'r.X.|
> 2a7bf9b30 72 f2 a7 7c 73 4f 2f a6 53 ef 02 14 5b 7c 93 9e |r..|sO/.S...[|..|
> 2aae14530 25 6b d5 ab b0 48 94 cd 53 ef 06 08 41 e0 0b ba |%k...H..S...A...|
> 2ac470a30 a4 a6 cb a0 40 a2 8f cd 53 ef 06 47 3d d5 e0 38 |....@...S..G=..8|
> 2aca3fb30 99 e4 1a cd b1 a8 be 6c 53 ef 01 03 68 6f 4c f4 |.......lS...hoL.|
> 2b19b6030 30 64 6a d9 08 f0 46 c9 53 ef 07 95 b2 14 fb aa |0dj...F.S.......|
> 2c53e0a30 59 1b 4b c8 25 a1 56 d9 53 ef 0b 26 3a 76 72 a5 |Y.K.%.V.S..&:vr.|
> 2cb48d730 0a d2 b4 d5 40 fc c5 d0 53 ef 0b 6d 90 9b db 22 |....@...S..m..."|
> 2dc925b30 16 18 5b 30 09 18 f8 2e 53 ef 0a d1 00 74 13 e7 |..[0....S....t..|
> 2dcc76230 e9 75 d0 2e 92 45 59 2c 53 ef 0e 84 63 9a aa b6 |.u...EY,S...c...|
> 2e6d28730 f4 81 47 6a 8f 9c ae 1a 53 ef 09 9d d2 5b e4 35 |..Gj....S....[.5|
> 2f9639f30 82 67 ae 49 2c a1 fc c0 53 ef 07 b1 1f 54 c5 c9 |.g.I,...S....T..|
> 2fa204230 78 42 16 12 e9 83 72 88 53 ef 00 08 df d9 8b 2a |xB....r.S......*|
> 31b7c1030 99 f0 43 99 1f 52 77 17 53 ef 0c 5f b9 51 a1 42 |..C..Rw.S.._.Q.B|
> 321ad8a30 4e 9b 9e 2e c6 8e 19 8d 53 ef 0e 1d 5d 20 c0 9d |N.......S...] ..|
> 32ff61630 72 9f 28 d2 9a 35 79 63 53 ef 09 e6 d6 e3 27 c5 |r.(..5ycS.....'.|
> 33a4b8230 58 9d 65 cb da 31 07 e7 53 ef 04 07 e2 4a 9b 17 |X.e..1..S....J..|
> 3472a7230 b4 a9 fa cf 3d 39 c3 95 53 ef 03 a4 07 2a ac 9a |....=9..S....*..|
> 3534f5130 54 d2 19 77 a4 c2 4a 2f 53 ef 07 b3 0f 60 be ee |T..w..J/S....`..|
> 356145130 df f7 f8 eb 6b 2a 8b fa 53 ef 09 cf 9d cd db 68 |....k*..S......h|
> 361b68830 53 ab 0c 7a 8e 4e 96 1b 53 ef 0f 3b 78 e9 d5 ce |S..z.N..S..;x...|
> 37381e630 24 63 4a ce 1b eb b0 df 53 ef 02 68 54 7d 7a ad |$cJ.....S..hT}z.|
> 375103330 3a 20 84 18 b6 6d f5 3b 53 ef 0a 86 8f ac b2 d8 |: ...m.;S.......|
> 37b3f7230 11 43 a4 46 fd c8 da ae 53 ef 04 f3 80 db 5c 4b |.C.F....S.....\K|
> 3a5adf530 7f c1 e1 75 26 cf 25 b2 53 ef 01 4d 64 71 10 bf |...u&.%.S..Mdq..|
> 3ccc7b830 55 cb 6f 68 e3 c0 0f 36 53 ef 06 00 85 8a 17 72 |U.oh...6S......r|
> 3d6743f30 fa ca 0e 36 a6 4e 02 6a 53 ef 05 6d 6f e5 31 78 |...6.N.jS..mo.1x|
> 3e8040030 0c 58 34 52 02 00 19 00 53 ef 01 00 01 00 00 00 |.X4R....S.......|
>
> ===============================================================================
Hi Phil,
Did you already have time to look at the results of the fsck check and
ext2/3/4 superblock search?
I would really like some feedback, as I'm quite out of ideas.
Thanks!
Olivier
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
