sudden failure of whole lvm system

Ciprian Pantea <cipixul@gmail.com> · Wed, 7 Dec 2005 19:50:08 +0200

Hello all!

I have a raid setup in stripping over some sata hdds(/dev/sda) and I setup lvm ontop of it.
now after one reboot I couldn't mount the lvm no more.

the /dev/mapper/ contains only the control device
dmesg |grep "device mapper":
    device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm-devel@redhat.com
pvs --verbose:
    Scanning for physical volume names
pvscan --verbose
    Wiping cache of LVM-capable devices
    Wiping internal cache
    Walking through all physical volumes
  No matching physical volumes found

pvdisplay: nothing

going to /etc/lvm/archive I see that I have some files with some configs and the latest one appears to be the good one:

# Generated by LVM2: Tue Jun 14 03:27:02 2005
contents = "Text Format Volume Group"
version = 1
description = "Created *before* executing 'lvcreate -l 1144408 archive'"
creation_host = "sakunew"       # Linux sakunew 2.6.11.12-arc-accpx #1 SMP Sun Jun 12 22:39:40 MST 2005 i686
creation_time = 1118744822      # Tue Jun 14 03:27:02 2005
archive {
        id = "I0Cbr4-o1RK-RclG-4gkI-zHca-12ac-1ASEOh"
        seqno = 1
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes

        max_lv = 0
        max_pv = 0
        physical_volumes {
                pv0 {
                        id = "cNr18P-wskQ-bvR3-FTON-1HVE-SotJ-bpFDhP"
                        device = "/dev/sda"     # Hint only
                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 1144408      # 4.36557 Terabytes
                }
        }
        logical_volumes {
                lvol0 {
                        id = "2Z4RG7-zWsp-wlg0-M7s5-GP4F-fGIF-8K5K4i"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1144408  # 4.36557 Terabytes
                                type = "striped"
                                stripe_count = 1        # linear
                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
        }
}

I tried the PV metadata recovery command found at 
http://www.tldp.org/HOWTO/LVM-HOWTO/recovermetadata.html
root@sakunew:/etc/lvm/archive# pvcreate --uuid "cNr18P-wskQ-bvR3-FTON-1HVE-SotJ-bpFDhP" --restorefile /etc/lvm/archive/archive_00003.vg -d /dev/sda
  Couldn't find device with uuid 'cNr18P-wskQ-bvR3-FTON-1HVE-SotJ-bpFDhP'.

  Device /dev/sda not found.

I tried also vgcfgrestore :
root@sakunew:/etc/lvm/archive# vgcfgrestore -d -v archive
  Couldn't find device with uuid 'cNr18P-wskQ-bvR3-FTON-1HVE-SotJ-bpFDhP'.
  Couldn't find all physical volumes for volume group archive.

  Restore failed.

when running cat /dev/sda | hexdump -C | less
I get :
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001c0  01 00 ee fe ff ff 01 00  00 00 ff ff ff ff 00 00  |..îþÿÿ....ÿÿÿÿ..|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............Uª|
00000200  45 46 49 20 50 41 52 54  00 00 01 00 5c 00 00 00  |EFI PART....\...|
00000210  65 36 d6 7e 00 00 00 00  01 00 00 00 00 00 00 00  |e6Ö~............|

00000220  ff 0f cb 2e 02 00 00 00  22 00 00 00 00 00 00 00  |ÿ.Ë.....".......|
00000230  de 0f cb 2e 02 00 00 00  4c de eb 10 ba 5e c6 4d  |Þ.Ë.....LÞë.º^ÆM|
00000240  98 a0 e4 a7 f3 9e 90 62  02 00 00 00 00 00 00 00  |. ä§ó..b........|

00000250  80 00 00 00 80 00 00 00  59 cc a7 59 00 00 00 00  |........YÌ§Y....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000a00  df aa fa b7 20 4c 56 4d  32 20 78 5b 35 41 25 72  |ßªú· LVM2 x[5A%r|

00000a10  30 4e 2a 3e 01 00 00 00  00 08 00 00 00 00 00 00  |0N*>............|
00000a20  00 f8 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |.ø..............|
00000a30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*
00004600  4c 41 42 45 4c 4f 4e 45  01 00 00 00 00 00 00 00  |LABELONE........|
00004610  8c 54 98 8d 20 00 00 00  4c 56 4d 32 20 30 30 31  |.T.. ...LVM2 001|
00004620  48 32 30 79 50 73 33 5a  30 4c 34 39 4e 58 59 46  |H20yPs3Z0L49NXYF|

00004630  71 64 4b 76 75 69 56 6d  5a 48 57 4c 7a 6e 32 57  |qdKvuiVmZHWLzn2W|
00004640  00 00 00 96 5d 04 00 00  00 00 03 00 00 00 00 00  |....]...........|
00004650  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

00004660  00 00 00 00 00 00 00 00  00 08 00 00 00 00 00 00  |................|
00004670  00 f8 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |.ø..............|
00004680  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*
00004c00  df aa fa b7 20 4c 56 4d  32 20 78 5b 35 41 25 72  |ßªú· LVM2 x[5A%r|
00004c10  30 4e 2a 3e 01 00 00 00  00 08 00 00 00 00 00 00  |0N*>............|
00004c20  00 f8 02 00 00 00 00 00  00 00 00 00 00 00 00 00  |.ø..............|

00004c30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004e00  61 72 63 68 69 76 65 20  7b 0a 69 64 20 3d 20 22  |archive {.id = "|
00004e10  71 74 72 52 68 61 2d 72  77 73 42 2d 67 54 39 4c  |qtrRha-rwsB-gT9L|

00004e20  2d 44 63 39 31 2d 75 52  70 62 2d 57 46 50 6e 2d  |-Dc91-uRpb-WFPn-|
00004e30  37 63 54 4a 66 71 22 0a  73 65 71 6e 6f 20 3d 20  |7cTJfq".seqno = |
00004e40  31 0a 73 74 61 74 75 73  20 3d 20 5b 22 52 45 53  |1.status = ["RES|

00004e50  49 5a 45 41 42 4c 45 22  2c 20 22 52 45 41 44 22  |IZEABLE", "READ"|
00004e60  2c 20 22 57 52 49 54 45  22 5d 0a 65 78 74 65 6e  |, "WRITE"].exten|
00004e70  74 5f 73 69 7a 65 20 3d  20 38 31 39 32 0a 6d 61  |t_size = 
8192.ma|
00004e80  78 5f 6c 76 20 3d 20 30  0a 6d 61 78 5f 70 76 20  |x_lv = 0.max_pv |
00004e90  3d 20 30 0a 0a 70 68 79  73 69 63 61 6c 5f 76 6f  |= 0..physical_vo|
00004ea0  6c 75 6d 65 73 20 7b 0a  0a 70 76 30 20 7b 0a 69  |lumes {..pv0 {.i|

00004eb0  64 20 3d 20 22 48 32 30  79 50 73 2d 33 5a 30 4c  |d = "H20yPs-3Z0L|
00004ec0  2d 34 39 4e 58 2d 59 46  71 64 2d 4b 76 75 69 2d  |-49NX-YFqd-Kvui-|
00004ed0  56 6d 5a 48 2d 57 4c 7a  6e 32 57 22 0a 64 65 76  |VmZH-WLzn2W".dev|

00004ee0  69 63 65 20 3d 20 22 2f  64 65 76 2f 73 64 61 31  |ice = "/dev/sda1|
00004ef0  22 0a 0a 73 74 61 74 75  73 20 3d 20 5b 22 41 4c  |"..status = ["AL|
00004f00  4c 4f 43 41 54 41 42 4c  45 22 5d 0a 70 65 5f 73  |LOCATABLE"].pe_s|

00004f10  74 61 72 74 20 3d 20 33  38 34 0a 70 65 5f 63 6f  |tart = 384.pe_co|
00004f20  75 6e 74 20 3d 20 31 31  34 34 34 30 38 0a 7d 0a  |unt = 1144408.}.|
00004f30  7d 0a 0a 7d 0a 23 20 47  65 6e 65 72 61 74 65 64  |}..}.# Generated|

00004f40  20 62 79 20 4c 56 4d 32  3a 20 54 75 65 20 4a 75  | by LVM2: Tue Ju|
00004f50  6e 20 31 34 20 30 33 3a  31 37 3a 32 35 20 32 30  |n 14 03:17:25 20|
00004f60  30 35 0a 0a 63 6f 6e 74  65 6e 74 73 20 3d 20 22  |05..contents = "|

00004f70  54 65 78 74 20 46 6f 72  6d 61 74 20 56 6f 6c 75  |Text Format Volu|
00004f80  6d 65 20 47 72 6f 75 70  22 0a 76 65 72 73 69 6f  |me Group".versio|
00004f90  6e 20 3d 20 31 0a 0a 64  65 73 63 72 69 70 74 69  |n = 1..descripti|

00004fa0  6f 6e 20 3d 20 22 22 0a  0a 63 72 65 61 74 69 6f  |on = ""..creatio|
00004fb0  6e 5f 68 6f 73 74 20 3d  20 22 73 61 6b 75 6e 65  |n_host = "sakune|
00004fc0  77 22 09 23 20 4c 69 6e  75 78 20 73 61 6b 75 6e  |w".# Linux sakun|

00004fd0  65 77 20 32 2e 36 2e 31  31 2e 31 32 2d 61 72 63  |ew 2.6.11.12-arc|
00004fe0  2d 61 63 63 70 78 20 23  31 20 53 4d 50 20 53 75  |-accpx #1 SMP Su|
00004ff0  6e 20 4a 75 6e 20 31 32  20 32 32 3a 33 39 3a 34  |n Jun 12 22:39:4|

00005000  30 20 4d 53 54 20 32 30  30 35 20 69 36 38 36 0a  |0 MST 2005 i686.|
00005010  63 72 65 61 74 69 6f 6e  5f 74 69 6d 65 20 3d 20  |creation_time = |
00005020  31 31 31 38 37 34 34 32  34 35 09 23 20 54 75 65  |1118744245.# Tue|

00005030  20 4a 75 6e 20 31 34 20  30 33 3a 31 37 3a 32 35  | Jun 14 03:17:25|
00005040  20 32 30 30 35 0a 0a 00  00 00 00 00 00 00 00 00  | 2005...........|
00005050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

*
00008200  4a 46 53 31 01 00 00 00  60 fe fc ff 00 00 00 00  |JFS1....`þüÿ....|
00008210  00 10 00 00 0c 00 03 00  00 02 00 00 09 00 00 00  |................|
00008220  00 00 40 00 00 09 20 10  02 00 00 00 00 00 00 00  |..@... .........|

00008230  04 00 00 00 53 00 01 00  02 00 00 00 51 00 01 00  |....S.......Q...|

indicating that both the archive UUID and the pv0 UUID changed and that pv0 now is on /dev/sda1(!) and that the JFS partition is there and possibly could be recovered ... 

I made a backup of the archive_00003.vg config and modified it with the data that I found in the hexdump above and now file looks like:

# Generated by LVM2: Tue Jun 14 03:27:02 2005
contents = "Text Format Volume Group"
version = 1
description = "Created *before* executing 'lvcreate -l 1144408 archive'"
creation_host = "sakunew"       # Linux sakunew 2.6.11.12-arc-accpx #1 SMP Sun Jun 12 22:39:40 MST 2005 i686
creation_time = 1118744822      # Tue Jun 14 03:27:02 2005
archive {
        id = "qtrRha-rwsB-gT9L-Dc91-uRpb-WFPn-7cTJfq"
        seqno = 1
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes

        max_lv = 0
        max_pv = 0
        physical_volumes {
                pv0 {
                        id = "H20yPs-3Z0L-49NX-YFqd-Kvui-VmZH-WLzn2W"
                        device = "/dev/sda1"    # Hint only
                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 1144408      # 4.36557 Terabytes
                }
        }
        logical_volumes {
                lvol0 {
                        id = "2Z4RG7-zWsp-wlg0-M7s5-GP4F-fGIF-8K5K4i"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 1144408  # 4.36557 Terabytes
                                type = "striped"
                                stripe_count = 1        # linear
                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
        }
}

now why my original config showed that pv0 was on /dev/sda and in metadata taken with hexdump it was in /dev/sda1 ?
if I run :
pvcreate --uuid "H20yPs-3Z0L-49NX-YFqd-Kvui-VmZH-WLzn2W" --restorefile /etc/lvm/archive/archive_00003.vg -d /dev/sda1
  Couldn't find device with uuid 'H20yPs-3Z0L-49NX-YFqd-Kvui-VmZH-WLzn2W'.
  Physical volume "/dev/sda1" successfully created

and 
root@sakunew:/etc/lvm/archive# pvscan
  PV /dev/sda1         lvm2 [2.00 TB]
  Total: 1 [2.00 TB] / in use: 0 [0   ] / in no VG: 1 [2.00 TB]

it shows only 2T instead of 4T which is in the config file
and 
root@sakunew:/etc/lvm/archive# pvdisplay
  --- NEW Physical volume ---
  PV Name               /dev/sda1
  VG Name
  PV Size               2.00 TB
  Allocatable           NO

  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               H20yPs-3Z0L-49NX-YFqd-Kvui-VmZH-WLzn2W

I'm running debian 3.1 with kernel 2.6.12 with 
ARECA RAID: 64BITS PCI BUS DMA ADDRESSING SUPPORTED
scsi0 : ARECA ARC1130 PCI-X 12 PORTS SATA RAID CONTROLLER (RAID6-ENGINE Inside)
        Driver Version 1.20.00.07
  Vendor: Areca     Model: ARC-1130-VOL#00   Rev: R001

  Type:   Direct-Access                      ANSI SCSI revision: 03
arcmsr device major number 253
SCSI device sda: 9374994432 512-byte hdwr sectors (4799997 MB)
SCSI device sda: drive cache: write back

what can I do to get lvm to detect my raid correctly and where is the problem so I can fix it and get my data ?
I hope I pasted all information that was needed. Now don't go ignoring me with this please ;p
thanks, cip

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/