Sorry for the long break away from this topic....
On Mar 7, 2012, at 2:31 PM, Lars Ellenberg wrote:
> On Mon, Mar 05, 2012 at 12:46:15PM -0600, Richard Petty wrote:
>> GOAL: Retrieve a KVM virtual machine from an inaccessible LVM volume.
>>
>> DESCRIPTION: In November, I was working on a home server. The system
>> boots to software mirrored drives but I have a hardware-based RAID5
>> array on it and I decided to create a logical volume and mount it at
>> /var/lib/libvirt/images so that all my KVM virtual machine image
>> files would reside on the hardware RAID.
>>
>> All that worked fine. Later, I decided to expand that
>> logical volume and that's when I made a mistake which wasn't
>> discovered until about six weeks later when I accidentally rebooted
>> the server. (Good problems usually require several mistakes.)
>>
>> Somehow, I accidentally mis-specified the second LMV physical
>> volume that I added to the volume group. When trying to activate
>> the LV filesystem, the device mapper now complains:
>>
>> LOG ENTRY
>> table: 253:3: sdc2 too small for target: start=2048, len=1048584192, dev_size=1048577586
>>
>> As you can see, the length is greater than the device size.
>>
>> I do not know how this could have happened. I assumed that LVM tool
>> sanity checking would have prevented this from happening.
>>
>> PV0 is okay.
>> PV1 is defective.
>> PV2 is okay but too small to receive a PV1's contents, I think.
>> PV3 was just added, hoping to migrate PV1 contents to it.
>>
>> So I added PV3 and tried to do a move but it seems that using some
>> of the LMV tools is predicated on the kernel being able to activate
>> everything, which it refuses to do.
>>
>> Can't migrate the data, can't resize anything. I'm stuck. If course
>> I've done a lot of Google research over the months but I have yet to
>> see a problem such as this solved.
>>
>> Got ideas?
>>
>> Again, my goal is to pluck a copy of a 100GB virtual machine off of
>> the LV. After that, I'll delete the LV.
>>
>> ==========================
>>
>> LMV REPORT FROM /etc/lvm/archive BEFORE THE CORRUPTION
>>
>> vg_raid {
>> id = "JLeyHJ-saON-6NSF-4Hqc-1rTA-vOWE-CU5aDZ"
>> seqno = 2
>> status = ["RESIZEABLE", "READ", "WRITE"]
>> flags = []
>> extent_size = 8192 # 4 Megabytes
>> max_lv = 0
>> max_pv = 0
>> metadata_copies = 0
>>
>> physical_volumes {
>>
>> pv0 {
>> id = "QaF9P6-Q9ch-bFTa-O3z2-3Idi-SdIw-YMLkQI"
>> device = "/dev/sdc1" # Hint only
>>
>> status = ["ALLOCATABLE"]
>> flags = []
>> dev_size = 419430400 # 200 Gigabytes
>> pe_start = 2048
>
> that's number of sectors into /dev/sdc1 "Hint only"
>
>> pe_count = 51199 # 199.996 Gigabytes
>> }
>> }
>>
>> logical_volumes {
>>
>> kvmfs {
>> id = "Hs636n-PLcl-aivI-VbTe-CAls-Zul8-m2liRY"
>> status = ["READ", "WRITE", "VISIBLE"]
>> flags = []
>> segment_count = 1
>>
>> segment1 {
>> start_extent = 0
>> extent_count = 50944 # 199 Gigabytes
>
> And that tells us your kvmfs lv is
> linear, not fragmented, and starting at extent 0.
> Which is, as seen above, 2048 sectors into sdc1.
>
> Try this, then look at /dev/mapper/maybe_kvmfs
> echo "0 $[50944 * 8192] linear /dev/sdc1 2048" |
> dmsetup create maybe_kvmfs
This did result in creating an entry at /dev/mapper/maybe_kvmfs.
> But see below...
>
>> type = "striped"
>> stripe_count = 1 # linear
>>
>> stripes = [
>> "pv0", 0
>> ]
>> }
>> }
>> }
>> }
>>
>> ==========================
>>
>> LMV REPORT FROM /etc/lvm/archive AS SEEN TODAY
>>
>> vg_raid {
>> id = "JLeyHJ-saON-6NSF-4Hqc-1rTA-vOWE-CU5aDZ"
>> seqno = 13
>> status = ["RESIZEABLE", "READ", "WRITE"]
>> flags = []
>> extent_size = 8192 # 4 Megabytes
>> max_lv = 0
>> max_pv = 0
>> metadata_copies = 0
>>
>> physical_volumes {
>>
>> pv0 {
>> id = "QaF9P6-Q9ch-bFTa-O3z2-3Idi-SdIw-YMLkQI"
>> device = "/dev/sdc1" # Hint only
>>
>> status = ["ALLOCATABLE"]
>> flags = []
>> dev_size = 419430400 # 200 Gigabytes
>> pe_start = 2048
>> pe_count = 51199 # 199.996 Gigabytes
>> }
>>
>> pv1 {
>> id = "8o0Igh-DKC8-gsof-FuZX-2Irn-qekz-0Y2mM9"
>> device = "/dev/sdc2" # Hint only
>>
>> status = ["ALLOCATABLE"]
>> flags = []
>> dev_size = 2507662218 # 1.16772 Terabytes
>> pe_start = 2048
>> pe_count = 306110 # 1.16772 Terabytes
>> }
>>
>> pv2 {
>> id = "NuW7Bi-598r-cnLV-E1E8-Srjw-4oM4-77RJkU"
>> device = "/dev/sdb5" # Hint only
>>
>> status = ["ALLOCATABLE"]
>> flags = []
>> dev_size = 859573827 # 409.877 Gigabytes
>> pe_start = 2048
>> pe_count = 104928 # 409.875 Gigabytes
>> }
>>
>> pv3 {
>> id = "eL40Za-g3aS-92Uc-E0fT-mHrP-5rO6-HT7pKK"
>> device = "/dev/sdc3" # Hint only
>>
>> status = ["ALLOCATABLE"]
>> flags = []
>> dev_size = 1459084632 # 695.746 Gigabytes
>> pe_start = 2048
>> pe_count = 178110 # 695.742 Gigabytes
>> }
>> }
>>
>> logical_volumes {
>>
>> kvmfs {
>> id = "Hs636n-PLcl-aivI-VbTe-CAls-Zul8-m2liRY"
>> status = ["READ", "WRITE", "VISIBLE"]
>> flags = []
>> segment_count = 2
>
> Oops, why does it have two segments now?
> That must have been your resize attempt.
>
>> segment1 {
>> start_extent = 0
>> extent_count = 51199 # 199.996 Gigabytes
>>
>> type = "striped"
>> stripe_count = 1 # linear
>>
>> stripes = [
>> "pv0", 0
>> ]
>> }
>> segment2 {
>> start_extent = 51199
>> extent_count = 128001 # 500.004 Gigabytes
>>
>> type = "striped"
>> stripe_count = 1 # linear
>>
>> stripes = [
>> "pv1", 0
>
> Fortunately simple again: two segments,
> both starting at extent 0 of their respective pv.
> that gives us:
>
> echo "0 $[51199 * 8192] linear /dev/sdc1 2048
> $[51199 * 8192] $[128001 * 8192] linear /dev/sdc2 2048" |
> dmsetup create maybe_kvmfs
>
> (now do some read-only sanity checks...)
I tried this command, decrementing sdc2 from 128001 to 127999:
[root@zeus /dev/mapper] echo "0 $[51199 * 8192] linear /dev/sdc1 2048 $[51199 * 8192] $[127999 * 8192] linear /dev/sdc2 2048" | dmsetup create kvmfs
device-mapper: create ioctl failed: Device or resource busy
Command failed
> Of course you need to adjust sdc1 and sdc2 to
> whatever is "right".
>
> According to the meta data dump above,
> "sdc1" is supposed to be your old 200 GB PV,
> and "sdc2" the 1.6 TB partition.
>
> The other PVs are "sdb5" (410 GB),
> and a "sdc3" of 695 GB...
>
> If 128001 is too large, reduce until it fits.
> If you broke the partition table,
> and the partition offsets are now wrong,
> you have to experiment a lot,
> and hope for the best.
>
> That will truncate the "kvmfs",
> but should not cause too much loss.
>
> If you figured out the correct PVs and offsets,
> you should be able to recover it all.
I understand that the strategy is to reduce the declared size of PV1 so that LVM can enable the PV and I can mount the kvmfs LV. I'm not expert at LVM, and while I can get some things done with it when there are no problems, I'm out of my league when problems occur.
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
