Andrei> I apologize for replying to my own message, I was subscribed in the Andrei> digest mode... No problem Andrei> I've been hitting my head against this one for a while Andrei> now. Originally discovered this on Ubuntu 20.04, but I'm Andrei> seeing the same issue with RHEL 8.5. The loss of a single disk Andrei> leaves the RAID in "partial" mode, when it should be Andrei> "degraded". Ouch, this isn't good. But why aren't you using MD RAID on top of the disks (partitioned ideally in my book) and then turn that MD device into a PV in a VG, and then make your LVs in there? Andrei> I've tried to explicitly specify the number of stripes, but it Andrei> did not make a difference. After adding the missing disk back, Andrei> the array is healthy again. Please see below. I'm wondering if there's something setup in the defaults for /etc/lvm.conf which makes a degraded array fail, instead of coming up degraded. But honestly, if you're looking for disk level redundancy, then I'd stronly recommend you use disks -> MD RAID6 -> LVM - > filesystem for your data. It's reliable, durable and well understood. I know there's an attraction to runnning LVs and RAID all together, since that should be easier to manage, right? But I think not. Have you tried to activate the LV using: lvchange -ay --activationmode degraded LV as a test? What does it say? I'm looking at the lvmraid man page for this suggestion. Andrei> # cat /etc/redhat-release Andrei> Red Hat Enterprise Linux release 8.5 (Ootpa) Andrei> # lvm version Andrei> LVM version: 2.03.12(2)-RHEL8 (2021-05-19) Andrei> Library version: 1.02.177-RHEL8 (2021-05-19) Andrei> Driver version: 4.43.0 Andrei> # lsblk Andrei> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT Andrei> sda 8:0 0 50G 0 disk Andrei> ├─sda1 8:1 0 1G 0 part /boot Andrei> └─sda2 8:2 0 49G 0 part Andrei> ├─rhel-root 253:0 0 44G 0 lvm / Andrei> └─rhel-swap 253:1 0 5G 0 lvm [SWAP] Andrei> sdb 8:16 0 70G 0 disk Andrei> sdc 8:32 0 100G 0 disk Andrei> sdd 8:48 0 100G 0 disk Andrei> sde 8:64 0 100G 0 disk Andrei> sdf 8:80 0 100G 0 disk Andrei> sdg 8:96 0 100G 0 disk Andrei> # pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg Andrei> # vgcreate pool_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3 Andrei> --stripesize 1 pool_vg Andrei> Invalid stripe size 1.00 KiB. Andrei> Run `lvcreate --help' for more information. Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3 Andrei> --stripesize 4 pool_vg Andrei> Logical volume "pool_lv" created. Andrei> # mkfs.xfs /dev/pool_vg/pool_lv Andrei> # echo "/dev/mapper/pool_vg-pool_lv /mnt xfs Andrei> defaults,x-systemd.mount-timeout=30 0 0" >> /etc/fstab Andrei> # mount -a Andrei> # touch /mnt/test Andrei> Note the RAID is correctly striped across all 5 disks: Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg Andrei> LV Attr Cpy%Sync Health Devices Andrei> pool_lv rwi-aor--- 100.00 Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0) Andrei> [pool_lv_rimage_0] iwi-aor--- /dev/sdc(1) Andrei> [pool_lv_rimage_1] iwi-aor--- /dev/sdd(1) Andrei> [pool_lv_rimage_2] iwi-aor--- /dev/sde(1) Andrei> [pool_lv_rimage_3] iwi-aor--- /dev/sdf(1) Andrei> [pool_lv_rimage_4] iwi-aor--- /dev/sdg(1) Andrei> [pool_lv_rmeta_0] ewi-aor--- /dev/sdc(0) Andrei> [pool_lv_rmeta_1] ewi-aor--- /dev/sdd(0) Andrei> [pool_lv_rmeta_2] ewi-aor--- /dev/sde(0) Andrei> [pool_lv_rmeta_3] ewi-aor--- /dev/sdf(0) Andrei> [pool_lv_rmeta_4] ewi-aor--- /dev/sdg(0) Andrei> After shutting down the OS and removing a disk, reboot drops the Andrei> system into a single user mode because it cannot mount /mnt! The RAID Andrei> is now in "partial" mode, when it must be just "degraded". Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg Andrei> WARNING: Couldn't find device with uuid Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co. Andrei> WARNING: VG pool_vg is missing PV Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co (last written to /dev/sdc). Andrei> LV Attr Cpy%Sync Health Devices Andrei> pool_lv rwi---r-p- partial Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0) Andrei> [pool_lv_rimage_0] Iwi---r-p- partial [unknown](1) Andrei> [pool_lv_rimage_1] Iwi---r--- /dev/sdc(1) Andrei> [pool_lv_rimage_2] Iwi---r--- /dev/sdd(1) Andrei> [pool_lv_rimage_3] Iwi---r--- /dev/sde(1) Andrei> [pool_lv_rimage_4] Iwi---r--- /dev/sdf(1) Andrei> [pool_lv_rmeta_0] ewi---r-p- partial [unknown](0) Andrei> [pool_lv_rmeta_1] ewi---r--- /dev/sdc(0) Andrei> [pool_lv_rmeta_2] ewi---r--- /dev/sdd(0) Andrei> [pool_lv_rmeta_3] ewi---r--- /dev/sde(0) Andrei> [pool_lv_rmeta_4] ewi---r--- /dev/sdf(0) Andrei> After adding the missing disk back, the system boots correctly and Andrei> there are no issues with the RAID: Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg Andrei> LV Attr Cpy%Sync Health Devices Andrei> pool_lv rwi-a-r--- 100.00 Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0) Andrei> [pool_lv_rimage_0] iwi-aor--- /dev/sdc(1) Andrei> [pool_lv_rimage_1] iwi-aor--- /dev/sdd(1) Andrei> [pool_lv_rimage_2] iwi-aor--- /dev/sde(1) Andrei> [pool_lv_rimage_3] iwi-aor--- /dev/sdf(1) Andrei> [pool_lv_rimage_4] iwi-aor--- /dev/sdg(1) Andrei> [pool_lv_rmeta_0] ewi-aor--- /dev/sdc(0) Andrei> [pool_lv_rmeta_1] ewi-aor--- /dev/sdd(0) Andrei> [pool_lv_rmeta_2] ewi-aor--- /dev/sde(0) Andrei> [pool_lv_rmeta_3] ewi-aor--- /dev/sdf(0) Andrei> [pool_lv_rmeta_4] ewi-aor--- /dev/sdg(0) Andrei> On Tue, Jan 25, 2022 at 11:44 AM Andrei Rodionov Andrei> <andrei.rodionov@xxxxxxxxx> wrote: >> >> Hello, >> >> I've provisioned an LVM RAID 6 across 5 physical disks. I'm trying to understand the RAID behavior after injecting the failure - removing physical disk /dev/sdc. Andrei> _______________________________________________ Andrei> linux-lvm mailing list Andrei> linux-lvm@xxxxxxxxxx Andrei> https://listman.redhat.com/mailman/listinfo/linux-lvm Andrei> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ _______________________________________________ linux-lvm mailing list linux-lvm@xxxxxxxxxx https://listman.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
