Good morning all,
After losing all power to our DC last night due to a storm, nearly all
of the volumes in our Pike cluster are unmountable. Of the 30 VMs in
use at the time, only one has been able to successfully mount and boot
from its rootfs. We are using Ceph as the backend storage to cinder
and glance. Any help or pointers to bring this back online would be
appreciated.
What most of the volumes are seeing is
[ 2.622252] SGI XFS with ACLs, security attributes, no debug enabled
[ 2.629285] XFS (sda1): Mounting V5 Filesystem
[ 2.832223] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[ 2.838412] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current]
[ 2.842383] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[ 2.846152] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 80 2c 19 00 04
00 00
[ 2.850146] blk_update_request: I/O error, dev sda, sector 8399897
or
[ 2.590178] EXT4-fs (vda1): INFO: recovery required on readonly
filesystem
[ 2.594319] EXT4-fs (vda1): write access will be enabled during recovery
[ 2.957742] print_req_error: I/O error, dev vda, sector 227328
[ 2.962468] Buffer I/O error on dev vda1, logical block 0, lost async
page write
[ 2.967933] Buffer I/O error on dev vda1, logical block 1, lost async
page write
[ 2.973076] print_req_error: I/O error, dev vda, sector 229384
As a test for one of the less critical vms, I deleted the vm and mounted
the volume on the one VM I managed to start. The results were not
promising:
# dmesg |tail
[ 5.136862] type=1305 audit(1530847244.811:4): audit_pid=496 old=0
auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
[ 7.726331] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[29374.967315] scsi 2:0:0:1: Direct-Access QEMU QEMU HARDDISK
2.5+ PQ: 0 ANSI: 5
[29374.988104] sd 2:0:0:1: [sdb] 83886080 512-byte logical blocks: (42.9
GB/40.0 GiB)
[29374.991126] sd 2:0:0:1: Attached scsi generic sg1 type 0
[29374.995302] sd 2:0:0:1: [sdb] Write Protect is off
[29374.997109] sd 2:0:0:1: [sdb] Mode Sense: 63 00 00 08
[29374.997186] sd 2:0:0:1: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[29375.005968] sdb: sdb1
[29375.007746] sd 2:0:0:1: [sdb] Attached SCSI disk
# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: QEMU QEMU HARDDISK (scsi)
Disk /dev/sdb: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 42.9GB 42.9GB primary xfs boot
# mount -t xfs /dev/sdb temp
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
# xfs_repair /dev/sdb
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!
attempting to find secondary superblock...
Which eventually fails. The ceph cluster looks healthy, I can export
the volumes from rbd. I can find no other errors in ceph of openstack
indicating a fault in either system.
- Is this recoverable?
- What happened to all of these volumes and can this be prevented
from occurring again? Note that any shutdown vm at the time of the
outage appears to be fine.
Relevant versions:
Base OS: all Centos 7.5
Ceph: Luminous 12.2.5-0
Openstack: Latest Pike releases in centos-release-openstack-pike-1-1
nova 16.1.4-1
cinder 11.1.1-1
--
Gary Molenkamp Computer Science/Science Technology Services
Systems Administrator University of Western Ontario
molenkam@xxxxxx http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
