I was pvmove-ing around data on a large (75TB), clustered (2-node) VG when the process hung:
[CHI (UTC+0000) root@dicombackup01 ~]# pvmove -b /dev/dm-10:0-2000
[CHI (UTC+0000) root@dicombackup01 ~]# pvmove -v -i600
Checking progress every 600 seconds
Finding all volume groups
Finding volume group "vg_store"
Finding volume group "vg00"
Finding volume group "vg_dicomstore"
/dev/dm-10: Moved: 0.1%
Executing: /sbin/modprobe dm-log-clustered
Updating volume group metadata
Error locking on node dicombackup01-int.chi.nighthawkrad.net: Command timed out
Failed to suspend lv_NRS_20090405
With no progress 8 hours later (and no IO to the relevant devices), I decided to kill the pvmove process and reboot the host. On reboot, starting clvmd kernel panicked the system. After messing around for some time - and setting locking = 0 in /etc/lvm/lvm.conf to avoid having to start up all the clustering infrastructure - I discovered that it was activating the VG that was causing the problem.
I assume that something has gotten corrupted in the metadata and is crashing the system when it is read. I have "archive" copies of the metadata that were created during the pvmove process. Ie:
[CHI (UTC+0000) root@dicombackup01 ~]# head -20 /etc/lvm/backup/vg_dicomstore
# Generated by LVM2 version 2.02.46-RHEL5 (2009-09-15): Sun Feb 7 08:18:09 2010
contents = "Text Format Volume Group"
version = 1
description = "Created *after* executing 'pvmove -b /dev/dm-10:0-2000'"
creation_host = "dicombackup01.chi.nighthawkrad.net" # Linux dicombackup01.chi.nighthawkrad.net 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64
creation_time = 1265530689 # Sun Feb 7 08:18:09 2010
vg_dicomstore {
id = "w7YvIp-bjYd-sNag-m0DD-t2fL-ShXd-dssXTY"
seqno = 2047
status = ["RESIZEABLE", "READ", "WRITE", "CLUSTERED"]
flags = []
extent_size = 2097152 # 1024 Megabytes
max_lv = 0
max_pv = 0
physical_volumes {
[CHI (UTC+0000) root@dicombackup01 ~]# head -20 /etc/lvm/archive/vg_dicomstore_00820.vg
# Generated by LVM2 version 2.02.46-RHEL5 (2009-09-15): Sun Feb 7 08:18:02 2010
contents = "Text Format Volume Group"
version = 1
description = "Created *before* executing 'pvmove -b /dev/dm-10:0-2000'"
creation_host = "dicombackup01.chi.nighthawkrad.net" # Linux dicombackup01.chi.nighthawkrad.net 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64
creation_time = 1265530682 # Sun Feb 7 08:18:02 2010
vg_dicomstore {
id = "w7YvIp-bjYd-sNag-m0DD-t2fL-ShXd-dssXTY"
seqno = 2046
status = ["RESIZEABLE", "READ", "WRITE", "CLUSTERED"]
flags = []
extent_size = 2097152 # 1024 Megabytes
max_lv = 0
max_pv = 0
physical_volumes {
Should I just try and manually overwrite the metadata on the PVs with /etc/lvm/archive/vg_dicomstore_00820.vg (the one created before running the last pvmove) ? Will that make the failed pvmove "disappear" (like a pvmove --abort would) ?
Any help would be greatly appreciated.
--
Christopher Smith
UNIX Team Leader
NightHawk Radiology Services
Suite 600, 4900 N Scottsdale Road
Scottsdale, 85251, USA
http://www.nighthawkrad.net
USA Toll free: 866 241 6635
Email: csmith@nighthawkrad.net
IP Extension: 4483
Sydney Mobile: +61 4 0739 7563
Sydney Phone: +61 2 8211 2363
US Mobile/Cell: +1 480 717 9562
US Phone: +1 480 822 4483
US Fax: +1 208 763 3643
All phones forwarded to my current location, however, please consider the local time in Arizona before calling from abroad.
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
