Malahal, And, after putting the fix referenced in the bug report in place, re-creating the initrd, I rebooted. Once back up, I took another look, and what I see is below. Success! I'll have to test it by yanking the fibre pairs again tomorrow, but it sure looks like we have conclusively found the culprit, and the solution. Many thanks to you, Malahal, and also to Chandra. You guys rock! root@zimbra-mb1:~# dmsetup table | egrep '^mb1_' mb1_t1-redo: 0 314572800 striped 4 128 254:4 99615104 254:8 99615104 254:2 99615104 254:1 99615104 mb1_t1-bin: 0 41943040 striped 4 128 254:4 384 254:8 384 254:2 384 254:1 384 mb1_t2-hsm_store3: 0 718585856 linear 254:6 24584576 mb1_t2-hsm_store3: 718585856 329990144 linear 254:7 384 mb1_t2-hsm_store2: 0 1023991808 linear 254:9 384 mb1_t2-hsm_store2: 1023991808 24584192 linear 254:6 384 mb1_t2-hsm_store1: 0 1048576000 linear 254:5 384 mb1_t1-store: 0 419430400 striped 4 128 254:4 178258304 254:8 178258304 254:2 178258304 254:1 178258304 mb1_t1-db: 0 104857600 striped 4 128 254:4 20971904 254:8 20971904 254:2 20971904 254:1 20971904 mb1_t1-log: 0 41943040 striped 4 128 254:4 10486144 254:8 10486144 254:2 10486144 254:1 10486144 mb1_t1-index: 0 209715200 striped 4 128 254:4 47186304 254:8 47186304 254:2 47186304 254:1 47186304 Cheers, -Brian Brian Elliott Finley wrote: > Look what I found! > > https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/246324 > > Trying the fix now... > > > Brian Elliott Finley wrote: >> Malahal, >> >> I took a look inside the initrd to see what it might be doing to >> contribute to this. What I see is the following: >> >> ./etc/udev/rules.d/85-lvm2.rules >> ./etc/udev/rules.d/95-multipath.rules >> >> Looks like a vgscan is being done inside the initrd (without an lvm.conf >> file) before multipath is setup. Could that be the cause? >> >> -Brian >> >> >> >> Brian Elliott Finley wrote: >>> Malahal, >>> >>> Good call. It changed. >>> >>> So I'm wondering why does it not re-associate with a vgscan? And why is >>> it associated thusly in the first place? >>> >>> root@zimbra-mb1:~# vgchange -an mb1_t1 >>> 0 logical volume(s) in volume group "mb1_t1" now active >>> root@zimbra-mb1:~# vgchange -ay mb1_t1 >>> 6 logical volume(s) in volume group "mb1_t1" now active >>> root@zimbra-mb1:~# dmsetup table | egrep '^mb1_t1' >>> mb1_t1-redo: 0 314572800 striped 4 128 254:6 99615104 254:9 99615104 >>> 254:22 99615104 254:23 99615104 >>> mb1_t1-bin: 0 41943040 striped 4 128 254:6 384 254:9 384 254:22 384 >>> 254:23 384 >>> mb1_t1-store: 0 419430400 striped 4 128 254:6 178258304 254:9 178258304 >>> 254:22 178258304 254:23 178258304 >>> mb1_t1-db: 0 104857600 striped 4 128 254:6 20971904 254:9 20971904 >>> 254:22 20971904 254:23 20971904 >>> mb1_t1-log: 0 41943040 striped 4 128 254:6 10486144 254:9 10486144 >>> 254:22 10486144 254:23 10486144 >>> mb1_t1-index: 0 209715200 striped 4 128 254:6 47186304 254:9 47186304 >>> 254:22 47186304 254:23 47186304 >>> >>> -Brian >>> >>> >>> malahal@us.ibm.com wrote: >>>> Brain, >>>> >>>> Your 'dmsetup table' output didn't seem to correspond to this 'lvs' >>>> command output. Your 'dmsetup table' command has this entry: >>>> >>>> mb1_t1-bin: 0 41943040 striped 4 128 8:0 384 8:16 384 8:32 384 8:48 384 >>>> >>>> That clearly says your 'mb1_t1/bin' volume is triped across devices with >>>> major:minor numbers as 8:0, 8:16, 8:32, 8:48. The major number 8 >>>> normally belongs to sd driver that controls single paths. Your 'lvs ..." >>>> output shows that it should be stripped across >>>> zimbra-mb1-t1-v1_fujitsu1-lun55 and others. >>>> >>>> Your "pvs" showed multipath PV names in your first email. I believe, >>>> something changed when you posted your "dmsetup ls" and "dmsetup table" >>>> commands. >>>> >>>> --Malahal. >>>> PS: Does your 'dmsetup table' now show the same output as before? >>>> >>>> >>>> Brian Elliott Finley [finley@anl.gov] wrote: >>>>> Malahal, >>>>> >>>>> Here is the output from the command you suggested (I'm sure I'll be >>>>> using that one again -- quite handy): >>>>> >>>>> root@zimbra-mb1:~# lvs -o lv_name,vg_name,devices >>>>> LV VG Devices >>>>> >>>>> >>>>> >>>>> bin mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(0),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(0),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(0),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(0) >>>>> >>>>> db mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(2560),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(2560),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(2560),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(2560) >>>>> >>>>> index mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(5760),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(5760),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(5760),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(5760) >>>>> >>>>> log mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(1280),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(1280),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(1280),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(1280) >>>>> >>>>> redo mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(12160),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(12160),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(12160),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(12160) >>>>> store mb1_t1 >>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55(21760),/dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56(21760),/dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57(21760),/dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58(21760) >>>>> hsm_store1 mb1_t2 /dev/mapper/zimbra-mb1-t2-v1_fujitsu1-lun63(0) >>>>> >>>>> >>>>> >>>>> hsm_store2 mb1_t2 /dev/mapper/zimbra-mb1-t2-v2_fujitsu1-lun64(0) >>>>> >>>>> >>>>> >>>>> hsm_store2 mb1_t2 /dev/mapper/zimbra-mb1-t2-v5_fujitsu1-lun67(0) >>>>> >>>>> >>>>> >>>>> hsm_store3 mb1_t2 >>>>> /dev/mapper/zimbra-mb1-t2-v5_fujitsu1-lun67(3001) >>>>> >>>>> >>>>> hsm_store3 mb1_t2 /dev/mapper/zimbra-mb1-t2-v6_fujitsu1-lun68(0) >>>>> >>>>> >>>>> >>>>> home zimbra-mb1 /dev/md1(832) >>>>> >>>>> >>>>> >>>>> root zimbra-mb1 /dev/md1(512) >>>>> >>>>> >>>>> >>>>> swap zimbra-mb1 /dev/md1(416) >>>>> >>>>> >>>>> >>>>> tmp zimbra-mb1 /dev/md1(1152) >>>>> >>>>> >>>>> >>>>> var zimbra-mb1 /dev/md1(1472) >>>>> >>>>> >>>>> -Brian >>>>> >>>>> >>>>> malahal@us.ibm.com wrote: >>>>>> Try running "lvs -o lv_name,vg_name,devices". It will list your logical >>>>>> volumes and their backing devices. I also found that your stripes are on >>>>>> single path devices as Chandra found. >>>>>> >>>>>> Maybe a problem with your filter??? >>>>>> >>>>>> --Malahal. >>>>>> >>>>>> Chandra Seetharaman [sekharan@us.ibm.com] wrote: >>>>>>> Hi Brian, >>>>>>> >>>>>>> which of these are your LVM volumes ? >>>>>>> >>>>>>> Perusal of "dmsetup table" shows you do not have your LVM volumes on top >>>>>>> of multipath devices. (I an elaborate on it once I get your answer for >>>>>>> the question above) >>>>>>> >>>>>>> chandra >>>>>>> On Wed, 2009-07-15 at 17:22 -0500, Brian E. Finley wrote: >>>>>>>> Chandra, >>>>>>>> >>>>>>>> root@zimbra-mb1:~# dmsetup ls >>>>>>>> mb1_t1-redo (254, 4) >>>>>>>> zimbra-mb1-t2-v6_fujitsu1-lun68 (254, 29) >>>>>>>> zimbra--mb1-home (254, 19) >>>>>>>> zimbra-mb1-t2-v5_fujitsu1-lun67 (254, 24) >>>>>>>> mb1_t1-bin (254, 0) >>>>>>>> zimbra-mb1-t2-v4_fujitsu1-lun66 (254, 6) >>>>>>>> zimbra--mb1-var (254, 21) >>>>>>>> mb1_t2-hsm_store3 (254, 12) >>>>>>>> zimbra-mb1-t2-v3_fujitsu1-lun65 (254, 28) >>>>>>>> zimbra-mb1-t2-v2_fujitsu1-lun64 (254, 26) >>>>>>>> mb1_t2-hsm_store2 (254, 10) >>>>>>>> zimbra--mb1-swap (254, 17) >>>>>>>> zimbra-mb1-t2-v1_fujitsu1-lun63 (254, 25) >>>>>>>> zimbra--mb1-root (254, 18) >>>>>>>> 35000c5000b36aa2b (254, 11) >>>>>>>> mb1_t2-hsm_store1 (254, 7) >>>>>>>> mb1_t1-store (254, 5) >>>>>>>> mb1_t1-db (254, 2) >>>>>>>> mb1_t1-log (254, 1) >>>>>>>> 35000c5000b15fe7b-part2 (254, 14) >>>>>>>> 35000c5000b15fe7b-part1 (254, 13) >>>>>>>> zimbra-mb1-t1-v4_fujitsu1-lun58 (254, 9) >>>>>>>> zimbra-mb1-t1-v3_fujitsu1-lun57 (254, 22) >>>>>>>> zimbra-mb1-t1-v2_fujitsu1-lun56 (254, 27) >>>>>>>> zimbra-mb1-t1-v1_fujitsu1-lun55 (254, 23) >>>>>>>> 35000c5000b36aa2b-part2 (254, 16) >>>>>>>> 35000c5000b15fe7b (254, 8) >>>>>>>> 35000c5000b36aa2b-part1 (254, 15) >>>>>>>> zimbra--mb1-tmp (254, 20) >>>>>>>> mb1_t1-index (254, 3) >>>>>>>> >>>>>>>> >>>>>>>> root@zimbra-mb1:~# dmsetup table >>>>>>>> mb1_t1-redo: 0 314572800 striped 4 128 8:0 99615104 8:16 99615104 8:32 99615104 8:48 99615104 >>>>>>>> zimbra-mb1-t2-v6_fujitsu1-lun68: 0 491911168 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 66:112 1000 8:144 1000 65:48 1000 65:208 1000 >>>>>>>> zimbra--mb1-home: 0 20971520 linear 9:1 54526336 >>>>>>>> zimbra-mb1-t2-v5_fujitsu1-lun67: 0 743178240 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 65:32 1000 8:128 1000 65:192 1000 66:96 1000 >>>>>>>> mb1_t1-bin: 0 41943040 striped 4 128 8:0 384 8:16 384 8:32 384 8:48 384 >>>>>>>> zimbra-mb1-t2-v4_fujitsu1-lun66: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:112 1000 65:16 1000 65:176 1000 66:80 1000 >>>>>>>> zimbra--mb1-var: 0 41943040 linear 9:1 96469376 >>>>>>>> mb1_t2-hsm_store3: 0 718585856 linear 65:192 24584576 >>>>>>>> mb1_t2-hsm_store3: 718585856 329990144 linear 65:208 384 >>>>>>>> zimbra-mb1-t2-v3_fujitsu1-lun65: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:96 1000 65:0 1000 65:160 1000 66:64 1000 >>>>>>>> zimbra-mb1-t2-v2_fujitsu1-lun64: 0 1024000000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 66:48 1000 8:80 1000 8:240 1000 65:144 1000 >>>>>>>> mb1_t2-hsm_store2: 0 1023991808 linear 8:240 384 >>>>>>>> mb1_t2-hsm_store2: 1023991808 24584192 linear 65:192 384 >>>>>>>> zimbra--mb1-swap: 0 6291456 linear 9:1 27263360 >>>>>>>> zimbra-mb1-t2-v1_fujitsu1-lun63: 0 1056964608 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:64 1000 8:224 1000 65:128 1000 66:32 1000 >>>>>>>> zimbra--mb1-root: 0 20971520 linear 9:1 33554816 >>>>>>>> 35000c5000b36aa2b: 0 286739329 multipath 0 0 1 1 round-robin 0 1 1 66:144 1000 >>>>>>>> mb1_t2-hsm_store1: 0 1048576000 linear 8:224 384 >>>>>>>> mb1_t1-store: 0 419430400 striped 4 128 8:0 178258304 8:16 178258304 8:32 178258304 8:48 178258304 >>>>>>>> mb1_t1-db: 0 104857600 striped 4 128 8:0 20971904 8:16 20971904 8:32 20971904 8:48 20971904 >>>>>>>> mb1_t1-log: 0 41943040 striped 4 128 8:0 10486144 8:16 10486144 8:32 10486144 8:48 10486144 >>>>>>>> 35000c5000b15fe7b-part2: 0 286326495 linear 254:8 401625 >>>>>>>> 35000c5000b15fe7b-part1: 0 401562 linear 254:8 63 >>>>>>>> zimbra-mb1-t1-v4_fujitsu1-lun58: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:48 1000 8:208 1000 65:112 1000 66:16 1000 >>>>>>>> zimbra-mb1-t1-v3_fujitsu1-lun57: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 65:96 1000 8:32 1000 8:192 1000 66:0 1000 >>>>>>>> zimbra-mb1-t1-v2_fujitsu1-lun56: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:176 1000 8:16 1000 65:80 1000 65:240 1000 >>>>>>>> zimbra-mb1-t1-v1_fujitsu1-lun55: 0 409600000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 4 1 8:0 1000 8:160 1000 65:64 1000 65:224 1000 >>>>>>>> 35000c5000b36aa2b-part2: 0 286326495 linear 254:11 401625 >>>>>>>> 35000c5000b15fe7b: 0 286739329 multipath 0 0 1 1 round-robin 0 1 1 66:128 1000 >>>>>>>> 35000c5000b36aa2b-part1: 0 401562 linear 254:11 63 >>>>>>>> zimbra--mb1-tmp: 0 20971520 linear 9:1 75497856 >>>>>>>> mb1_t1-index: 0 209715200 striped 4 128 8:0 47186304 8:16 47186304 8:32 47186304 8:48 47186304 >>>>>>>> >>>>>>>> >>>>>>>> Thanks, -Brian >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>> From: "Chandra Seetharaman" <sekharan@us.ibm.com> >>>>>>>> To: "Brian Elliott Finley" <finley@anl.gov>, "LVM general discussion and development" <linux-lvm@redhat.com> >>>>>>>> Sent: Wednesday, July 15, 2009 5:11:47 PM GMT -06:00 US/Canada Central >>>>>>>> Subject: Re: multipath works; lvm on multipath does not >>>>>>>> >>>>>>>> Can you send the o/p of "dmsetup ls" and "dmsetup table" >>>>>>>> >>>>>>>> On Wed, 2009-07-15 at 16:53 -0500, finley@anl.gov wrote: >>>>>>>>> How can I tell what is causing LVM on multipath volumes to fail, while direct multipath volumes are working? >>>>>>>>> >>>>>>>>> >>>>>>>>> The details: >>>>>>>>> >>>>>>>>> I have several volumes that are multipathed over fibre channel (2 qlogic cards in the host) from a Fujitsu Eternus 4000 (4 controllers). >>>>>>>>> >>>>>>>>> When using a file system mounted on the dm-multipath device directly (Ie.: generating I/O via IOZone), it fails over gracefully when I pull out either of the fibre pairs, and the file system continues to operate. This device is called /dev/mapper/zimbra-mb1-t2-v4_fujitsu1-lun66. >>>>>>>>> >>>>>>>>> However, when I access an LVM volume created with sister devices as PVs, the LVM presented volume does not fail over. Rather the LV device goes inaccessible and the file system (ext3) re-mounts itself as read-only. If I try to access any of the underlying PVs, they still respond as available (Ie.: fdisk -l $PV). >>>>>>>>> >>>>>>>>> I am not seeing duplicate devices, and am using the following filter in /etc/lvm/lvm.conf: >>>>>>>>> >>>>>>>>> filter = [ "a|^/dev/md.*|", "a|^/dev/mapper/zimbra-mb.*-t.*-v.*_.*-lun.*|", "r|.*|" ] >>>>>>>>> >>>>>>>>> >>>>>>>>> I have also added the following, but dm-multipath devices seem usable by LVM with or without this setting: >>>>>>>>> >>>>>>>>> types = [ "device-mapper", 1 ] >>>>>>>>> >>>>>>>>> >>>>>>>>> I have tried removing the /etc/lvm/cache/.cache file, but that seems to have had no effect. I've also tried re-building the initrd after modifying the lvm.conf file, also with no effect. >>>>>>>>> >>>>>>>>> Additional info: >>>>>>>>> root@zimbra-mb1:~# pvs >>>>>>>>> PV VG Fmt Attr PSize PFree >>>>>>>>> /dev/mapper/zimbra-mb1-t1-v1_fujitsu1-lun55 mb1_t1 lvm2 a- 195.31G 60.31G >>>>>>>>> /dev/mapper/zimbra-mb1-t1-v2_fujitsu1-lun56 mb1_t1 lvm2 a- 195.31G 60.31G >>>>>>>>> /dev/mapper/zimbra-mb1-t1-v3_fujitsu1-lun57 mb1_t1 lvm2 a- 195.31G 60.31G >>>>>>>>> /dev/mapper/zimbra-mb1-t1-v4_fujitsu1-lun58 mb1_t1 lvm2 a- 195.31G 60.31G >>>>>>>>> /dev/mapper/zimbra-mb1-t2-v1_fujitsu1-lun63 mb1_t2 lvm2 a- 504.00G 4.00G >>>>>>>>> /dev/mapper/zimbra-mb1-t2-v2_fujitsu1-lun64 mb1_t2 lvm2 a- 488.28G 0 >>>>>>>>> /dev/mapper/zimbra-mb1-t2-v3_fujitsu1-lun65 mb1_t2 lvm2 a- 195.31G 195.31G >>>>>>>>> /dev/mapper/zimbra-mb1-t2-v5_fujitsu1-lun67 mb1_t2 lvm2 a- 354.37G 0 >>>>>>>>> /dev/mapper/zimbra-mb1-t2-v6_fujitsu1-lun68 mb1_t2 lvm2 a- 234.56G 77.21G >>>>>>>>> /dev/md1 zimbra-mb1 lvm2 a- 136.50G 83.50G >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> kernel version: 2.6.24-24-server >>>>>>>>> distoro: Ubuntu Hardy >>>>>>>>> multipath-tools: 0.4.8-7ubuntu2 >>>>>>>>> lvm2: 2.02.26-1ubuntu9 >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> -Brian >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> linux-lvm mailing list >>>>>>>>> linux-lvm@redhat.com >>>>>>>>> https://www.redhat.com/mailman/listinfo/linux-lvm >>>>>>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>>>>> _______________________________________________ >>>>>>> linux-lvm mailing list >>>>>>> linux-lvm@redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/linux-lvm >>>>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>>>> _______________________________________________ >>>>>> linux-lvm mailing list >>>>>> linux-lvm@redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/linux-lvm >>>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>>> -- >>>>> Brian Elliott Finley >>>>> Deputy Manager, Unix, Storage, and Operations >>>>> Computing and Information Systems >>>>> Argonne National Laboratory >>>>> Office: 630.252.4742 >>>>> Mobile: 630.447.9108 >>>>> >>>>> _______________________________________________ >>>>> linux-lvm mailing list >>>>> linux-lvm@redhat.com >>>>> https://www.redhat.com/mailman/listinfo/linux-lvm >>>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ >>>> _______________________________________________ >>>> linux-lvm mailing list >>>> linux-lvm@redhat.com >>>> https://www.redhat.com/mailman/listinfo/linux-lvm >>>> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > -- Brian Elliott Finley Deputy Manager, Unix, Storage, and Operations Computing and Information Systems Argonne National Laboratory Office: 630.252.4742 Mobile: 630.447.9108 _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
