Original post possibly lost in moderation, this is a repost with more details and updated information. Linux hilfy 4.6.0-1-amd64 #1 SMP Debian 4.6.4-1a~test (2016-09-16) x86_64 GNU/Linux (essentially stock Debian kernel) LVM version: 2.02.164(2) (2016-08-15) Library version: 1.02.133 (2016-08-15) Driver version: 4.34.0 I have a 5 disk setup. There is one large PV per disk and a single VG. LVs are mostly raid5 and raid6. I needed to replace one disk so I researched and tested in a VM (debian-8.6.0-amd64-netinst.iso) first. Web searching LVM RAID repair was not too encouraging. The following recipe worked in the VM: Shutdown. Remove one virtual disk. Add new empty virtual disk. Boot. partition pvcreate, vgextend 3x lvconvert --repair LV vgreduce --removemissing I tested this with raid1, raid5 and raid6 LVs spanning all the PVs. It worked just fine. I came to do the same with a physical disk change. Take backups. Shutdown Remove one failing disk. Add new empty disk. Boot. partition, pcreate, vgextend I first repaired the root volume and this worked just fine. Then I tried to repair the var volume. I very unfortunately did not keep the error messages (but see below). They included a complaint about "new device injected into existing array without 'rebuild' parameter specified" and ended with a failure to lock something. Based on later research the output was probably: WARNING: Failed to connect to lvmetad. Falling back to device scanning. Couldn't find device with uuid [UUID]. Attempt to replace failed RAID images (requires full device resync)? [y/n]: Accepted input: [y] device-mapper: reload ioctl on ([major:minor]) failed: Invalid argument Failed to lock logical volume vg0/[LV]. Failed to replace faulty devices in vg0/[LV]. Various tests showed that /var was still functioning fine (and there appeared to have been a resync, possibly after I re-ran the command or touched the var FS). It was a bit disturbing to see duplicate symlinks (-> ../dm-$n) in /dev/mapper, but each duplicate was either a _missing or [inactive] device. The duplicates are no longer there (at least two reboots after repairing var and one after repairing home). I thought there might be some odd state in-kernel after the first repair; this is what prompted me to try repairing the next LV after a reboot. The net effect is that I now have RAID volumes in an odd state with extracted / missing images (I think just a cosmetic issue) and which rebuild on every boot (or rather re-activation and use). The volumes take a long time to resync, especially concurrently! I've verified parity (and syndrome) manually - the rebuilds are good (on to the replacement disk) and no data appear to have been lost or corrupted. The problem appears to be with the metadata. I've since tried to repair a fourth RAID volume and collected detailed diagnostics. The obvious symptoms are visible in lvs output and in /dev/mapper below. I've included much relevant information here: https://drive.google.com/drive/folders/0B8dHrWSoVcaDN0RHTjFjQjdQNW8 This includes kernel logs and full -vvvdddddd output and error of some commands, plus kernel logs as well as rmeta subLVs and a few utilities I wrote. Please let me know if any more information would be useful or whether this all belongs in a bug tracker somewhere. Thanks for any assistance. I really hope I can get this working properly again. --gp Earlier excerpt from lvs -a (I've since lvremove'd the extracted LVs and tried repairing photos): root vg0 rwi-aor--- 25.00g 100.00 [root_rimage_0] vg0 iwi-aor--- 6.25g [root_rimage_1] vg0 iwi-aor--- 6.25g [root_rimage_2] vg0 iwi-aor--- 6.25g [root_rimage_3] vg0 iwi-aor--- 6.25g [root_rimage_4] vg0 iwi-aor--- 6.25g [root_rmeta_0] vg0 ewi-aor--- 4.00m [root_rmeta_1] vg0 ewi-aor--- 4.00m [root_rmeta_2] vg0 ewi-aor--- 4.00m [root_rmeta_3] vg0 ewi-aor--- 4.00m [root_rmeta_4] vg0 ewi-aor--- 4.00m var vg0 rwi-a-r--- 25.00g 100.00 var_rimage_0__extracted vg0 -wi-----p- 6.25g [var_rimage_1] vg0 iwi-aor--- 6.25g [var_rimage_2] vg0 iwi-aor--- 6.25g [var_rimage_3] vg0 iwi-aor--- 6.25g [var_rimage_4] vg0 iwi-aor--- 6.25g [var_rimage_5] vg0 iwi-aor--- 6.25g var_rmeta_0__extracted vg0 -wi-----p- 4.00m [var_rmeta_1] vg0 ewi-aor--- 4.00m [var_rmeta_2] vg0 ewi-aor--- 4.00m [var_rmeta_3] vg0 ewi-aor--- 4.00m [var_rmeta_4] vg0 ewi-aor--- 4.00m [var_rmeta_5] vg0 ewi-aor--- 4.00m home vg0 rwi-aor--- 400.01g 100.00 home_rimage_0__extracted vg0 -wi-----p- 133.34g [home_rimage_1] vg0 iwi-aor--- 133.34g [home_rimage_2] vg0 iwi-aor--- 133.34g [home_rimage_3] vg0 iwi-aor--- 133.34g [home_rimage_4] vg0 iwi-aor--- 133.34g [home_rimage_5] vg0 iwi-aor--- 133.34g home_rmeta_0__extracted vg0 -wi-----p- 4.00m [home_rmeta_1] vg0 ewi-aor--- 4.00m [home_rmeta_2] vg0 ewi-aor--- 4.00m [home_rmeta_3] vg0 ewi-aor--- 4.00m [home_rmeta_4] vg0 ewi-aor--- 4.00m [home_rmeta_5] vg0 ewi-aor--- 4.00m photos vg0 rwi-aor-p- 600.00g 100.00 [photos_rimage_0] vg0 iwi-a-r-p- 200.00g [photos_rimage_1] vg0 iwi-aor--- 200.00g [photos_rimage_2] vg0 iwi-aor--- 200.00g [photos_rimage_3] vg0 iwi-aor--- 200.00g [photos_rimage_4] vg0 iwi-aor--- 200.00g [photos_rmeta_0] vg0 ewi-a-r-p- 4.00m [photos_rmeta_1] vg0 ewi-aor--- 4.00m [photos_rmeta_2] vg0 ewi-aor--- 4.00m [photos_rmeta_3] vg0 ewi-aor--- 4.00m [photos_rmeta_4] vg0 ewi-aor--- 4.00m Earlier excerpt from ls -l /dev/mapper: lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-root -> ../dm-10 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rimage_0 -> ../dm-1 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rimage_1 -> ../dm-3 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rimage_2 -> ../dm-5 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rimage_3 -> ../dm-7 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rimage_4 -> ../dm-9 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rmeta_0 -> ../dm-0 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rmeta_1 -> ../dm-2 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rmeta_2 -> ../dm-4 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rmeta_3 -> ../dm-6 lrwxrwxrwx 1 root root 7 Sep 24 01:27 vg0-root_rmeta_4 -> ../dm-8 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var -> ../dm-21 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rimage_1 -> ../dm-14 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rimage_2 -> ../dm-16 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rimage_3 -> ../dm-18 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rimage_4 -> ../dm-20 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rimage_5 -> ../dm-12 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rmeta_1 -> ../dm-13 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rmeta_2 -> ../dm-15 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rmeta_3 -> ../dm-17 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rmeta_4 -> ../dm-19 lrwxrwxrwx 1 root root 8 Sep 24 02:19 vg0-var_rmeta_5 -> ../dm-11 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home -> ../dm-32 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rimage_1 -> ../dm-25 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rimage_2 -> ../dm-27 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rimage_3 -> ../dm-29 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rimage_4 -> ../dm-31 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rimage_5 -> ../dm-23 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rmeta_1 -> ../dm-24 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rmeta_2 -> ../dm-26 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rmeta_3 -> ../dm-28 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rmeta_4 -> ../dm-30 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-home_rmeta_5 -> ../dm-22 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos -> ../dm-45 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_0 -> ../dm-36 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_0-missing_0_0 -> ../dm-35 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_1 -> ../dm-38 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_2 -> ../dm-40 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_3 -> ../dm-42 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rimage_4 -> ../dm-44 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_0 -> ../dm-34 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_0-missing_0_0 -> ../dm-33 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_1 -> ../dm-37 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_2 -> ../dm-39 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_3 -> ../dm-41 lrwxrwxrwx 1 root root 8 Sep 24 01:27 vg0-photos_rmeta_4 -> ../dm-43 _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
