What could this be symptoms of: kern.err<3>: Jan 28 15:26:34 Openfiler2 kernel: device-mapper: table: 253:1: linear: dm-linear: Device lookup failed kern.warn<4>: Jan 28 15:26:34 Openfiler2 kernel: device-mapper: ioctl: error adding target to table I'm not sure if this is more information than what I have below, or just more condensed and readable: [root@Openfiler2 ~]# pvs PV VG Fmt Attr PSize PFree /dev/dm-0 vg0drbd lvm2 a- 1.81t 0 /dev/sdb2 localvg lvm2 a- 975.44g 100.00g /dev/sdc1 localvg lvm2 a- 975.93g 0 [root@Openfiler2 ~]# vgs VG #PV #LV #SN Attr VSize VFree localvg 2 1 0 wz--n- 1.91t 100.00g vg0drbd 1 1 0 wz--n- 1.81t 0 [root@Openfiler2 ~]# lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert r1 localvg -wi-ao 1.81t filer vg0drbd -wi-d- 1.81t I see there is 100g free in localvg so maybe I will try increasing the size of vg vg0drbd and see if that helps me mount the volume... Kind regards, Seb A > -----Original Message----- > From: linux-lvm-bounces@redhat.com > [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Seb A > 4jngrhkk3c-at-snkmail.com |linux-lvm/Allow + Forward to > Syn&Yahoo&Gmail| > Sent: 28 January 2014 22:54 > To: ****************; ********************; > *****************; linux-lvm@redhat.com > Subject: Full DRDB device on LVM is now unusable > > I'm not sure if this is more of an LVM issue or a DRDB issue, > but maybe someone here can help me... > > My DRDB device on LVM filled up with data and now it is > unusable after a power cycle. The DRDB device that was not > on LVM is fine (but it did not fill up). > > I configured two DRDB nodes running Openfiler with corosync > and pacemaker as per the instructions here > [http://www.howtoforge.com/openfiler-2.99-active-passive-with- > corosync-pacemaker-and-drbd] over two years ago. At one > point it swapped over to what was originally the secondary > node "Openfiler2" and I left it like that and all was fine > (AFAIK). (I did have a few issues in the early days with it > losing sync on reboot / power failure, but that's ancient > history.) Eventually the DRBD data partition filled up as > there are processes that ftp files onto it. There were lots > of proftpd processes that were stuck trying to do a CWD into > the data partition and therefore the cpu 'load' went really > high. I tried to start a process to delete old files and it > got stuck. It wasn't doing anything, and I couldn't cancel > or kill it. kill -9 <pid> did not work on that process or > any of the stuck proftpd processes. So I could not unmount > the drive and when I tried fuser it just killed m! > y ssh session and failed to kill the proftpd processes. I > restarted sshd via the console, and, as there had been some > kernel panics I decided to reboot, hardly expecting it to > succeed. It didn't - it got stuck and I had to kill the > virtual power. When it came back up it could not mount the > DRBD data partition (that uses LVM). Both the DRBD > partitions were synchronized before and after the reboot - > they reconnected and the primary stayed on 'Openfiler2'. > > The first errors after the reboot were in here: > > daemon.info<30>: Jan 28 15:26:34 Openfiler2 LVM[3228]: INFO: > Activating volume group vg0drbd > daemon.info<30>: Jan 28 15:26:34 Openfiler2 LVM[3228]: INFO: > Reading all physical volumes. This may take a while... Found > volume group "localvg" using metadata type lvm2 Found volume > group "vg0drbd" using metadata type lvm2 > kern.err<3>: Jan 28 15:26:34 Openfiler2 kernel: > device-mapper: table: 253:1: linear: dm-linear: Device lookup failed > kern.warn<4>: Jan 28 15:26:34 Openfiler2 kernel: > device-mapper: ioctl: error adding target to table > daemon.err<27>: Jan 28 15:26:34 Openfiler2 LVM[3228]: ERROR: > device-mapper: reload ioctl failed: Invalid argument 1 > logical volume(s) in volume group "vg0drbd" now active > daemon.info<30>: Jan 28 15:26:34 Openfiler2 crmd: [1284]: > info: process_lrm_event: LRM operation lvmdata_start_0 > (call=26, rc=1, cib-update=31, confirmed=true) unknown error > > I tried to mount it manually but the device is missing. Any > suggestions on how I can get this volume mounted? Thanks! > > For reference: > kern.info<6>: Jan 28 17:28:11 Openfiler2 kernel: > device-mapper: uevent: version 1.0.3 > kern.info<6>: Jan 28 17:28:11 Openfiler2 kernel: > device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: > dm-devel@redhat.com > > Linux Openfiler2 2.6.32-71.18.1.el6-0.20.smp.gcc4.1.x86_64 #1 > SMP Fri Mar 25 23:12:47 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux > > My last line of /etc/fstab is commented out as it is > controlled by pacemaker: > #/dev/vg0drbd/filer /mnt/vg0drbd/filer xfs > defaults,usrquota,grpquota 0 0 > > > Right now I have: > > [root@Openfiler2 ~]# service drbd status > drbd driver loaded OK; device status: > version: 8.3.10 (api:88/proto:86-96) > GIT-hash: 5c0b0469666682443d4785d90a2c603378f9017b build by > phil@fat-tyre, 2011-01-28 12:17:35 > m:res cs ro ds > p mounted fstype > 0:cluster_metadata Connected Primary/Secondary > UpToDate/UpToDate C /cluster_metadata ext3 > 1:vg0_drbd Connected Primary/Secondary UpToDate/UpToDate C > > [root@Openfiler2 ~]# crm status > ============ > Last updated: Tue Jan 28 20:11:19 2014 > Stack: openais > Current DC: Openfiler1 - partition with quorum > Version: 1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b > 2 Nodes configured, 2 expected votes > 2 Resources configured. > ============ > > Online: [ Openfiler1 Openfiler2 ] > > Resource Group: g_services > MetaFS (ocf::heartbeat:Filesystem): Started Openfiler2 > lvmdata (ocf::heartbeat:LVM): Stopped > DataFS (ocf::heartbeat:Filesystem): Stopped > openfiler (lsb:openfiler): Stopped > ClusterIP (ocf::heartbeat:IPaddr2): Stopped > iscsi (lsb:iscsi-target): Stopped > ldap (lsb:ldap): Stopped > samba (lsb:smb): Stopped > nfs (lsb:nfs): Stopped > nfslock (lsb:nfslock): Stopped > ftp (lsb:proftpd): Stopped > Master/Slave Set: ms_g_drbd > Masters: [ Openfiler2 ] > Slaves: [ Openfiler1 ] > > Failed actions: > lvmdata_start_0 (node=Openfiler2, call=28, rc=1, > status=complete): unknown error > > ###### > > More reference: > > [root@Openfiler2 ~]# pvdisplay > --- Physical volume --- > PV Name /dev/sdc1 > VG Name localvg > PV Size 975.93 GiB / not usable 2.32 MiB > Allocatable yes (but full) > PE Size 4.00 MiB > Total PE 249837 > Free PE 0 > Allocated PE 249837 > PV UUID OPFfsk-LXkz-3Voc-CQbj-Qf8d-YmHs-cR4Xjt > > --- Physical volume --- > PV Name /dev/sdb2 > VG Name localvg > PV Size 975.44 GiB / not usable 3.32 MiB > Allocatable yes > PE Size 4.00 MiB > Total PE 249712 > Free PE 25600 > Allocated PE 224112 > PV UUID yG1gfI-1HRb-AdCS-RqUV-Cm2j-pdqe-ZcB10j > > --- Physical volume --- > PV Name /dev/dm-0 > VG Name vg0drbd > PV Size 1.81 TiB / not usable 1.11 MiB > Allocatable yes (but full) > PE Size 4.00 MiB > Total PE 473934 > Free PE 0 > Allocated PE 473934 > PV UUID u8Au1m-U1pJ-RMik-bZGk-7NPA-3EOL-P21MHW > > [root@Openfiler2 ~]# pvscan > PV /dev/sdc1 VG localvg lvm2 [975.93 GiB / 0 free] > PV /dev/sdb2 VG localvg lvm2 [975.44 GiB / 100.00 > GiB free] > PV /dev/localvg/r1 VG vg0drbd lvm2 [1.81 TiB / 0 free] > Total: 3 [3.71 TiB] / in use: 3 [3.71 TiB] / in no VG: 0 [0 ] > > [root@Openfiler2 ~]# vgdisplay > --- Volume group --- > VG Name localvg > System ID > Format lvm2 > Metadata Areas 2 > Metadata Sequence No 23 > VG Access read/write > VG Status resizable > MAX LV 0 > Cur LV 1 > Open LV 1 > Max PV 0 > Cur PV 2 > Act PV 2 > VG Size 1.91 TiB > PE Size 4.00 MiB > Total PE 499549 > Alloc PE / Size 473949 / 1.81 TiB > Free PE / Size 25600 / 100.00 GiB > VG UUID 5knbwX-LaJ5-1fEd-OD1R-59jZ-Otmy-8IKtVl > > --- Volume group --- > VG Name vg0drbd > System ID > Format lvm2 > Metadata Areas 1 > Metadata Sequence No 7 > VG Access read/write > VG Status resizable > MAX LV 0 > Cur LV 1 > Open LV 0 > Max PV 0 > Cur PV 1 > Act PV 1 > VG Size 1.81 TiB > PE Size 4.00 MiB > Total PE 473934 > Alloc PE / Size 473934 / 1.81 TiB > Free PE / Size 0 / 0 > VG UUID 4pgyVr-Eduj-2CVD-rUhf-Sr7L-Q814-45BE2N > > [root@Openfiler2 ~]# vgscan > Reading all physical volumes. This may take a while... > Found volume group "localvg" using metadata type lvm2 > Found volume group "vg0drbd" using metadata type lvm2 > > [root@Openfiler2 ~]# lvdisplay > --- Logical volume --- > LV Name /dev/localvg/r1 > VG Name localvg > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > LV Write Access read/write > LV Status available > # open 2 > LV Size 1.81 TiB > Current LE 473949 > Segments 2 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:0 > > --- Logical volume --- > LV Name /dev/vg0drbd/filer > VG Name vg0drbd > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > LV Write Access read/write > LV Status NOT available > LV Size 1.81 TiB > Current LE 473934 > Segments 1 > Allocation inherit > Read ahead sectors auto > > [root@Openfiler2 ~]# lvscan > ACTIVE '/dev/localvg/r1' [1.81 TiB] inherit > inactive '/dev/vg0drbd/filer' [1.81 TiB] inherit > > [root@Openfiler2 ~]# lvchange -ay /dev/vg0drbd/filer > device-mapper: reload ioctl failed: Invalid argument > > [root@Openfiler2 ~]# lvdisplay > --- Logical volume --- > LV Name /dev/localvg/r1 > VG Name localvg > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > LV Write Access read/write > LV Status available > # open 2 > LV Size 1.81 TiB > Current LE 473949 > Segments 2 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:0 > > --- Logical volume --- > LV Name /dev/vg0drbd/filer > VG Name vg0drbd > LV UUID eSuNJr-yFDC-WCET-sIgi-IgTf-JRYz-Ack7oe > LV Write Access read/write > LV Status available > # open 0 > LV Size 1.81 TiB > Current LE 473934 > Segments 1 > Allocation inherit > Read ahead sectors auto > - currently set to 256 > Block device 253:1 > > [root@Openfiler2 ~]# lvscan > ACTIVE '/dev/localvg/r1' [1.81 TiB] inherit > ACTIVE '/dev/vg0drbd/filer' [1.81 TiB] inherit > > [root@Openfiler2 ~]# ls -l /dev/dm-* > brw-rw---- 1 root disk 253, 0 Jan 28 17:39 /dev/dm-0 > brw-rw---- 1 root disk 253, 1 Jan 28 21:00 /dev/dm-1 > > [root@Openfiler2 ~]# dmsetup ls > localvg-r1 (253, 0) > vg0drbd-filer (253, 1) > > [root@Openfiler2 ~]# dmsetup info > Name: localvg-r1 > State: ACTIVE > Read Ahead: 256 > Tables present: LIVE > Open count: 2 > Event number: 0 > Major, minor: 253, 0 > Number of targets: 2 > UUID: > LVM-5knbwXLaJ51fEdOD1R59jZOtmy8IKtVleSuNJryFDCWCETsIgiIgTfJRYzAck7oe > > Name: vg0drbd-filer > State: ACTIVE > Read Ahead: 256 > Tables present: None > Open count: 0 > Event number: 0 > Major, minor: 253, 1 > Number of targets: 0 > UUID: > LVM-4pgyVrEduj2CVDrUhfSr7LQ81445BE2NeSuNJryFDCWCETsIgiIgTfJRYzAck7oe > > [root@Openfiler2 ~]# dmsetup deps > localvg-r1: 2 dependencies : (8, 18) (8, 33) > vg0drbd-filer: 0 dependencies : > > [root@Openfiler2 ~]# dmsetup table > localvg-r1: 0 2046664704 linear 8:33 2048 > localvg-r1: 2046664704 1835925504 linear 8:18 2048 > vg0drbd-filer: > > [root@Openfiler2 ~]# drbdsetup /dev/drbd1 show > disk { > size 0s _is_default; # bytes > on-io-error detach; > fencing dont-care _is_default; > max-bio-bvecs 0 _is_default; > } > net { > timeout 60 _is_default; # 1/10 seconds > max-epoch-size 2048 _is_default; > max-buffers 2048 _is_default; > unplug-watermark 128 _is_default; > connect-int 10 _is_default; # seconds > ping-int 10 _is_default; # seconds > sndbuf-size 0 _is_default; # bytes > rcvbuf-size 0 _is_default; # bytes > ko-count 0 _is_default; > after-sb-0pri disconnect _is_default; > after-sb-1pri disconnect _is_default; > after-sb-2pri disconnect _is_default; > rr-conflict disconnect _is_default; > ping-timeout 5 _is_default; # 1/10 seconds > on-congestion block _is_default; > congestion-fill 0s _is_default; # byte > congestion-extents 127 _is_default; > } > syncer { > rate 112640k; # bytes/second > after 0; > al-extents 127 _is_default; > on-no-data-accessible io-error _is_default; > c-plan-ahead 0 _is_default; # 1/10 seconds > c-delay-target 10 _is_default; # 1/10 seconds > c-fill-target 0s _is_default; # bytes > c-max-rate 102400k _is_default; # bytes/second > c-min-rate 4096k _is_default; # bytes/second > } > protocol C; > _this_host { > device minor 1; > disk "/dev/localvg/r1"; > meta-disk internal; > address ipv4 192.168.100.159:7789; > } > _remote_host { > address ipv4 192.168.100.158:7789; > } > > [root@Openfiler2 ~]# crm configure show > node Openfiler1 \ > attributes standby="off" > node Openfiler2 \ > attributes standby="off" > primitive ClusterIP ocf:heartbeat:IPaddr2 \ > params ip="192.168.4.157" cidr_netmask="32" \ > op monitor interval="30s" > primitive DataFS ocf:heartbeat:Filesystem \ > params device="/dev/vg0drbd/filer" > directory="/mnt/vg0drbd/filer" fstype="xfs" \ > meta target-role="started" > primitive MetaFS ocf:heartbeat:Filesystem \ > params device="/dev/drbd0" > directory="/cluster_metadata" fstype="ext3" \ > meta target-role="started" > primitive drbd_data ocf:linbit:drbd \ > params drbd_resource="vg0_drbd" \ > op monitor interval="15s" > primitive drbd_meta ocf:linbit:drbd \ > params drbd_resource="cluster_metadata" \ > op monitor interval="15s" > primitive ftp lsb:proftpd \ > meta target-role="stopped" > primitive iscsi lsb:iscsi-target > primitive ldap lsb:ldap > primitive lvmdata ocf:heartbeat:LVM \ > params volgrpname="vg0drbd" \ > meta target-role="started" > primitive nfs lsb:nfs > primitive nfslock lsb:nfslock > primitive openfiler lsb:openfiler > primitive samba lsb:smb > group g_drbd drbd_meta drbd_data > group g_services MetaFS lvmdata DataFS openfiler ClusterIP > iscsi ldap samba nfs nfslock ftp > ms ms_g_drbd g_drbd \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" clone-max="2" clone-node-max="1" notify="true" > location cli-prefer-ClusterIP ClusterIP \ > rule $id="cli-prefer-rule-ClusterIP" inf: #uname eq Openfiler1 > location cli-standby-g_services g_services \ > rule $id="cli-standby-rule-g_services" -inf: #uname > eq Openfiler1 > location cli-standby-ms_g_drbd ms_g_drbd \ > rule $id="cli-standby-ms_g_drbd-rule" $role="Master" > -inf: #uname eq Openfiler1 > colocation c_g_services_on_g_drbd inf: g_services ms_g_drbd:Master > order o_g_servicesafter_g_drbd inf: ms_g_drbd:promote g_services:start > property $id="cib-bootstrap-options" \ > dc-version="1.1.2-c6b59218ee949eebff30e837ff6f3824ed0ab86b" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="2" \ > stonith-enabled="false" \ > no-quorum-policy="ignore" \ > last-lrm-refresh="1390944138" > rsc_defaults $id="rsc-options" \ > resource-stickiness="100" > > I intentially stopped proftpd (ftp) via the Linux Cluster > Management Console 1.5.14 so that I didn't get more proftpd > processes starting up if you are wondering why it says stopped above. > > Many thanks and regards, > > Seb A > > _______________________________________________ > linux-lvm mailing list > linux-lvm@redhat.com > https://www.redhat.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ > _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
