Write Protect appears to be off here (Thanks Angel): [root@Openfiler2 ~]# grep -i "Write Protect" /var/log/messages kern.info<6>: Jan 28 17:28:11 Openfiler2 kernel: Write protecting the kernel read-only data: 7056k kern.notice<5>: Jan 28 17:28:11 Openfiler2 kernel: sd 0:0:0:0: [sda] Write Protect is off kern.notice<5>: Jan 28 17:28:11 Openfiler2 kernel: sd 0:0:1:0: [sdb] Write Protect is off kern.notice<5>: Jan 28 17:28:11 Openfiler2 kernel: sd 0:0:2:0: [sdc] Write Protect is off Kind regards, Seb A > -----Original Message----- > From: linux-lvm-bounces@redhat.com > [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Seb A > 4jngrhkk3c-at-snkmail.com |linux-lvm/Allow + Forward to > Syn&Yahoo&Gmail| > Sent: 29 January 2014 12:16 > To: ****************; ********************; > *****************; linux-lvm@redhat.com > Subject: Re: Full DRDB device on LVM is now unusable > > What could this be symptoms of: > > kern.err<3>: Jan 28 15:26:34 Openfiler2 kernel: > device-mapper: table: 253:1: > linear: dm-linear: Device lookup failed > kern.warn<4>: Jan 28 15:26:34 Openfiler2 kernel: > device-mapper: ioctl: error > adding target to table > > > I'm not sure if this is more information than what I have > below, or just > more condensed and readable: > > [root@Openfiler2 ~]# pvs > PV VG Fmt Attr PSize PFree > /dev/dm-0 vg0drbd lvm2 a- 1.81t 0 > /dev/sdb2 localvg lvm2 a- 975.44g 100.00g > /dev/sdc1 localvg lvm2 a- 975.93g 0 > > [root@Openfiler2 ~]# vgs > VG #PV #LV #SN Attr VSize VFree > localvg 2 1 0 wz--n- 1.91t 100.00g > vg0drbd 1 1 0 wz--n- 1.81t 0 > > [root@Openfiler2 ~]# lvs > LV VG Attr LSize Origin Snap% Move Log Copy% Convert > r1 localvg -wi-ao 1.81t > filer vg0drbd -wi-d- 1.81t > > I see there is 100g free in localvg so maybe I will try > increasing the size > of vg vg0drbd and see if that helps me mount the volume... > > Kind regards, > > Seb A > > > > > > -----Original Message----- > > From: linux-lvm-bounces@redhat.com > > [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Seb A > > 4jngrhkk3c-at-snkmail.com |linux-lvm/Allow + Forward to > > Syn&Yahoo&Gmail| > > Sent: 28 January 2014 22:54 > > To: ****************; ********************; > > *****************; linux-lvm@redhat.com > > Subject: Full DRDB device on LVM is now unusable > > > > I'm not sure if this is more of an LVM issue or a DRDB issue, > > but maybe someone here can help me... > > > > My DRDB device on LVM filled up with data and now it is > > unusable after a power cycle. The DRDB device that was not > > on LVM is fine (but it did not fill up). > > > > I configured two DRDB nodes running Openfiler with corosync > > and pacemaker as per the instructions here > > [http://www.howtoforge.com/openfiler-2.99-active-passive-with- > > corosync-pacemaker-and-drbd] over two years ago. At one > > point it swapped over to what was originally the secondary > > node "Openfiler2" and I left it like that and all was fine > > (AFAIK). (I did have a few issues in the early days with it > > losing sync on reboot / power failure, but that's ancient > > history.) Eventually the DRBD data partition filled up as > > there are processes that ftp files onto it. There were lots > > of proftpd processes that were stuck trying to do a CWD into > > the data partition and therefore the cpu 'load' went really > > high. I tried to start a process to delete old files and it > > got stuck. It wasn't doing anything, and I couldn't cancel > > or kill it. kill -9 <pid> did not work on that process or > > any of the stuck proftpd processes. So I could not unmount > > the drive and when I tried fuser it just killed m! > > y ssh session and failed to kill the proftpd processes. I > > restarted sshd via the console, and, as there had been some > > kernel panics I decided to reboot, hardly expecting it to > > succeed. It didn't - it got stuck and I had to kill the > > virtual power. When it came back up it could not mount the > > DRBD data partition (that uses LVM). Both the DRBD > > partitions were synchronized before and after the reboot - > > they reconnected and the primary stayed on 'Openfiler2'. > > > > The first errors after the reboot were in here: > > > > daemon.info<30>: Jan 28 15:26:34 Openfiler2 LVM[3228]: INFO: > > Activating volume group vg0drbd > > daemon.info<30>: Jan 28 15:26:34 Openfiler2 LVM[3228]: INFO: > > Reading all physical volumes. This may take a while... Found > > volume group "localvg" using metadata type lvm2 Found volume > > group "vg0drbd" using metadata type lvm2 > > kern.err<3>: Jan 28 15:26:34 Openfiler2 kernel: > > device-mapper: table: 253:1: linear: dm-linear: Device lookup failed > > kern.warn<4>: Jan 28 15:26:34 Openfiler2 kernel: > > device-mapper: ioctl: error adding target to table > > daemon.err<27>: Jan 28 15:26:34 Openfiler2 LVM[3228]: ERROR: > > device-mapper: reload ioctl failed: Invalid argument 1 > > logical volume(s) in volume group "vg0drbd" now active > > daemon.info<30>: Jan 28 15:26:34 Openfiler2 crmd: [1284]: > > info: process_lrm_event: LRM operation lvmdata_start_0 > > (call=26, rc=1, cib-update=31, confirmed=true) unknown error > > > > I tried to mount it manually but the device is missing. Any > > suggestions on how I can get this volume mounted? Thanks! > > > > For reference: > > kern.info<6>: Jan 28 17:28:11 Openfiler2 kernel: > > device-mapper: uevent: version 1.0.3 > > kern.info<6>: Jan 28 17:28:11 Openfiler2 kernel: > > device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: > > dm-devel@redhat.com > > > > Linux Openfiler2 2.6.32-71.18.1.el6-0.20.smp.gcc4.1.x86_64 #1 > > SMP Fri Mar 25 23:12:47 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > > > > My last line of /etc/fstab is commented out as it is > > controlled by pacemaker: > > #/dev/vg0drbd/filer /mnt/vg0drbd/filer xfs > > defaults,usrquota,grpquota 0 0 > > > > > > Right now I have: > > > > [root@Openfiler2 ~]# service drbd status > > drbd driver loaded OK; device status: > > version: 8.3.10 (api:88/proto:86-96) > > GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by > > phil@fat-tyre, 2011-01-28 12:17:35 > > m:res cs ro ds > > p mounted fstype > > 0:cluster_metadata Connected Primary/Secondary > > UpToDate/UpToDate C /cluster_metadata ext3 > > 1:vg0_drbd Connected Primary/Secondary > UpToDate/UpToDate C > > > > [root@Openfiler2 ~]# crm status > > ============ > > Last updated: Tue Jan 28 20:11:19 2014 > > Stack: openais > > Current DC: Openfiler1 - partition with quorum > > Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b > > 2 Nodes configured, 2 expected votes > > 2 Resources configured. > > ============ > > > > Online: [ Openfiler1 Openfiler2 ] > > > > Resource Group: g_services > > MetaFS (ocf::heartbeat:Filesystem): Started Openfiler2 > > lvmdata (ocf::heartbeat:LVM): Stopped > > DataFS (ocf::heartbeat:Filesystem): Stopped > > openfiler (lsb:openfiler): Stopped > > ClusterIP (ocf::heartbeat:IPaddr2): Stopped > > iscsi (lsb:iscsi-target): Stopped > > ldap (lsb:ldap): Stopped > > samba (lsb:smb): Stopped > > nfs (lsb:nfs): Stopped > > nfslock (lsb:nfslock): Stopped > > ftp (lsb:proftpd): Stopped > > Master/Slave Set: ms_g_drbd > > Masters: [ Openfiler2 ] > > Slaves: [ Openfiler1 ] > > > > Failed actions: > > lvmdata_start_0 (node=Openfiler2, call=28, rc=1, > > status=complete): unknown error > > > > ###### > > > > More reference: > > > > [root@Openfiler2 ~]# pvdisplay > > --- Physical volume --- > > PV Name /dev/sdc1 > > VG Name localvg > > PV Size 975.93 GiB / not usable 2.32 MiB > > Allocatable yes (but full) > > PE Size 4.00 MiB > > Total PE 249837 > > Free PE 0 > > Allocated PE 249837 > > PV UUID OPFfsk-LXkz-3Voc-CQbj-Qf8d-YmHs-cR4Xjt > > > > --- Physical volume --- > > PV Name /dev/sdb2 > > VG Name localvg > > PV Size 975.44 GiB / not usable 3.32 MiB > > Allocatable yes > > PE Size 4.00 MiB > > Total PE 249712 > > Free PE 25600 > > Allocated PE 224112 > > PV UUID yG1gfI-1HRb-AdCS-RqUV-Cm2j-pdqe-ZcB10j > > > > --- Physical volume --- > > PV Name /dev/dm-0 > > VG Name vg0drbd > > PV Size 1.81 TiB / not usable 1.11 MiB > > Allocatable yes (but full) > > PE Size 4.00 MiB > > Total PE 473934 > > Free PE 0 > > Allocated PE 473934 > > PV UUID u8Au1m-U1pJ-RMik-bZGk-7NPA-3EOL-P21MHW > > > > [root@Openfiler2 ~]# pvscan > > PV /dev/sdc1 VG localvg lvm2 [975.93 GiB / 0 free] > > PV /dev/sdb2 VG localvg lvm2 [975.44 GiB / 100.00 > > GiB free] > > PV /dev/localvg/r1 VG vg0drbd lvm2 [1.81 TiB / 0 free] > > Total: 3 [3.71 TiB] / in use: 3 [3.71 TiB] / in no VG: 0 [0 ] > > > > [root@Openfiler2 ~]# vgdisplay > > --- Volume group --- > > VG Name localvg > > System ID > > Format lvm2 > > Metadata Areas 2 > > Metadata Sequence No 23 > > VG Access read/write > > VG Status resizable > > MAX LV 0 > > Cur LV 1 > > Open LV 1 > > Max PV 0 > > Cur PV 2 > > Act PV 2 > > VG Size 1.91 TiB > > PE Size 4.00 MiB > > Total PE 499549 > > Alloc PE / Size 473949 / 1.81 TiB > > Free PE / Size 25600 / 100.00 GiB > > VG UUID 5knbwX-LaJ5-1fEd-OD1R-59jZ-Otmy-8IKtVl > > > > --- Volume group --- > > VG Name vg0drbd > > System ID > > Format lvm2 > > Metadata Areas 1 > > Metadata Sequence No 7 > > VG Access read/write > > VG Status resizable > > MAX LV 0 > > Cur LV 1 > > Open LV 0 > > Max PV 0 > > Cur PV 1 > > Act PV 1 > > VG Size 1.81 TiB > > PE Size 4.00 MiB > > Total PE 473934 > > Alloc PE / Size 473934 / 1.81 TiB > > Free PE / Size 0 / 0 > > VG UUID 4pgyVr-Eduj-2CVD-rUhf-Sr7L-Q814-45BE2N > > > > [root@Openfiler2 ~]# vgscan > > Reading all physical volumes. This may take a while... > > Found volume group "localvg" using metadata type lvm2 > > Found volume group "vg0drbd" using metadata type lvm2 > > > > [root@Openfiler2 ~]# lvdisplay > > --- Logical volume --- > > LV Name /dev/localvg/r1 > > VG Name localvg > > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > > LV Write Access read/write > > LV Status available > > # open 2 > > LV Size 1.81 TiB > > Current LE 473949 > > Segments 2 > > Allocation inherit > > Read ahead sectors auto > > - currently set to 256 > > Block device 253:0 > > > > --- Logical volume --- > > LV Name /dev/vg0drbd/filer > > VG Name vg0drbd > > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > > LV Write Access read/write > > LV Status NOT available > > LV Size 1.81 TiB > > Current LE 473934 > > Segments 1 > > Allocation inherit > > Read ahead sectors auto > > > > [root@Openfiler2 ~]# lvscan > > ACTIVE '/dev/localvg/r1' [1.81 TiB] inherit > > inactive '/dev/vg0drbd/filer' [1.81 TiB] inherit > > > > [root@Openfiler2 ~]# lvchange -ay /dev/vg0drbd/filer > > device-mapper: reload ioctl failed: Invalid argument > > > > [root@Openfiler2 ~]# lvdisplay > > --- Logical volume --- > > LV Name /dev/localvg/r1 > > VG Name localvg > > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > > LV Write Access read/write > > LV Status available > > # open 2 > > LV Size 1.81 TiB > > Current LE 473949 > > Segments 2 > > Allocation inherit > > Read ahead sectors auto > > - currently set to 256 > > Block device 253:0 > > > > --- Logical volume --- > > LV Name /dev/vg0drbd/filer > > VG Name vg0drbd > > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > > LV Write Access read/write > > LV Status available > > # open 0 > > LV Size 1.81 TiB > > Current LE 473934 > > Segments 1 > > Allocation inherit > > Read ahead sectors auto > > - currently set to 256 > > Block device 253:1 > > > > [root@Openfiler2 ~]# lvscan > > ACTIVE '/dev/localvg/r1' [1.81 TiB] inherit > > ACTIVE '/dev/vg0drbd/filer' [1.81 TiB] inherit > > > > [root@Openfiler2 ~]# ls -l /dev/dm-* > > brw-rw---- 1 root disk 253, 0 Jan 28 17:39 /dev/dm-0 > > brw-rw---- 1 root disk 253, 1 Jan 28 21:00 /dev/dm-1 > > > > [root@Openfiler2 ~]# dmsetup ls > > localvg-r1 (253, 0) > > vg0drbd-filer (253, 1) > > > > [root@Openfiler2 ~]# dmsetup info > > Name: localvg-r1 > > State: ACTIVE > > Read Ahead: 256 > > Tables present: LIVE > > Open count: 2 > > Event number: 0 > > Major, minor: 253, 0 > > Number of targets: 2 > > UUID: > > LVM-5knbwXLaJ51fEdOD1R59jZOtmy8IKtVleSuNJryFDCWCETsIgiIgTfJRYzAck7oe > > > > Name: vg0drbd-filer > > State: ACTIVE > > Read Ahead: 256 > > Tables present: None > > Open count: 0 > > Event number: 0 > > Major, minor: 253, 1 > > Number of targets: 0 > > UUID: > > LVM-4pgyVrEduj2CVDrUhfSr7LQ81445BE2NeSuNJryFDCWCETsIgiIgTfJRYzAck7oe > > > > [root@Openfiler2 ~]# dmsetup deps > > localvg-r1: 2 dependencies : (8, 18) (8, 33) > > vg0drbd-filer: 0 dependencies : > > > > [root@Openfiler2 ~]# dmsetup table > > localvg-r1: 0 2046664704 linear 8:33 2048 > > localvg-r1: 2046664704 1835925504 linear 8:18 2048 > > vg0drbd-filer: > > > > [root@Openfiler2 ~]# drbdsetup /dev/drbd1 show > > disk { > > size 0s _is_default; # bytes > > on-io-error detach; > > fencing dont-care _is_default; > > max-bio-bvecs 0 _is_default; > > } > > net { > > timeout 60 _is_default; # 1/10 seconds > > max-epoch-size 2048 _is_default; > > max-buffers 2048 _is_default; > > unplug-watermark 128 _is_default; > > connect-int 10 _is_default; # seconds > > ping-int 10 _is_default; # seconds > > sndbuf-size 0 _is_default; # bytes > > rcvbuf-size 0 _is_default; # bytes > > ko-count 0 _is_default; > > after-sb-0pri disconnect _is_default; > > after-sb-1pri disconnect _is_default; > > after-sb-2pri disconnect _is_default; > > rr-conflict disconnect _is_default; > > ping-timeout 5 _is_default; # 1/10 seconds > > on-congestion block _is_default; > > congestion-fill 0s _is_default; # byte > > congestion-extents 127 _is_default; > > } > > syncer { > > rate 112640k; # bytes/second > > after 0; > > al-extents 127 _is_default; > > on-no-data-accessible io-error _is_default; > > c-plan-ahead 0 _is_default; # 1/10 seconds > > c-delay-target 10 _is_default; # 1/10 seconds > > c-fill-target 0s _is_default; # bytes > > c-max-rate 102400k _is_default; # bytes/second > > c-min-rate 4096k _is_default; # bytes/second > > } > > protocol C; > > _this_host { > > device minor 1; > > disk "/dev/localvg/r1"; > > meta-disk internal; > > address ipv4 192.168.100.159:7789; > > } > > _remote_host { > > address ipv4 192.168.100.158:7789; > > } > > > > [root@Openfiler2 ~]# crm configure show > > node Openfiler1 \ > > attributes standby="off" > > node Openfiler2 \ > > attributes standby="off" > > primitive ClusterIP ocf:heartbeat:IPaddr2 \ > > params ip="192.168.4.157" cidr_netmask="32" \ > > op monitor interval="30s" > > primitive DataFS ocf:heartbeat:Filesystem \ > > params device="/dev/vg0drbd/filer" > > directory="/mnt/vg0drbd/filer" fstype="xfs" \ > > meta target-role="started" > > primitive MetaFS ocf:heartbeat:Filesystem \ > > params device="/dev/drbd0" > > directory="/cluster_metadata" fstype="ext3" \ > > meta target-role="started" > > primitive drbd_data ocf:linbit:drbd \ > > params drbd_resource="vg0_drbd" \ > > op monitor interval="15s" > > primitive drbd_meta ocf:linbit:drbd \ > > params drbd_resource="cluster_metadata" \ > > op monitor interval="15s" > > primitive ftp lsb:proftpd \ > > meta target-role="stopped" > > primitive iscsi lsb:iscsi-target > > primitive ldap lsb:ldap > > primitive lvmdata ocf:heartbeat:LVM \ > > params volgrpname="vg0drbd" \ > > meta target-role="started" > > primitive nfs lsb:nfs > > primitive nfslock lsb:nfslock > > primitive openfiler lsb:openfiler > > primitive samba lsb:smb > > group g_drbd drbd_meta drbd_data > > group g_services MetaFS lvmdata DataFS openfiler ClusterIP > > iscsi ldap samba nfs nfslock ftp > > ms ms_g_drbd g_drbd \ > > meta master-max="1" master-node-max="1" clone-max="2" > > clone-node-max="1" clone-max="2" clone-node-max="1" notify="true" > > location cli-prefer-ClusterIP ClusterIP \ > > rule $id="cli-prefer-rule-ClusterIP" inf: #uname eq > Openfiler1 > > location cli-standby-g_services g_services \ > > rule $id="cli-standby-rule-g_services" -inf: #uname > > eq Openfiler1 > > location cli-standby-ms_g_drbd ms_g_drbd \ > > rule $id="cli-standby-ms_g_drbd-rule" $role="Master" > > -inf: #uname eq Openfiler1 > > colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master > > order o_g_servicesafter_g_drbd inf: ms_g_drbd:promote > g_services:start > > property $id="cib-bootstrap-options" \ > > > dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \ > > cluster-infrastructure="openais" \ > > expected-quorum-votes="2" \ > > stonith-enabled="false" \ > > no-quorum-policy="ignore" \ > > last-lrm-refresh="1390944138" > > rsc_defaults $id="rsc-options" \ > > resource-stickiness="100" > > > > I intentially stopped proftpd (ftp) via the Linux Cluster > > Management Console 1.5.14 so that I didn't get more proftpd > > processes starting up if you are wondering why it says > stopped above. > > > > Many thanks and regards, > > > > Seb A > > > > _______________________________________________ > > linux-lvm mailing list > > linux-lvm@redhat.com > > https://www.redhat.com/mailman/listinfo/linux-lvm > > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > > > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
