During reorganization of the Ceph system, including an updated CRUSH map and moving to btrfs, some PGs became stuck incomplete+remapped. Before that was resolved, a restart of osd.1 failed while creating a btrfs snapshot. A 'ceph-osd -i 1 --flush-journal' fails with the same error. See the below pasted log. This is a Bad Thing, because two PGs are now stuck down+peering. A 'ceph pg 2.74 query' shows they had been stuck on osd.1 before the btrfs problem, despite what the 'last acting' field shows in the below 'ceph health detail' output. Is there any way to recover from this? Judging from Google searches on the list archives, nobody has run into this problem before, so I'm quite worried that this spells backup recovery exercises for the next few days. Related question: Are outright OSD crashes the reason btrfs is discouraged for production use? Thanks- John pg 2.74 is stuck inactive since forever, current state down+peering, last acting [3,7,0,6] pg 3.73 is stuck inactive since forever, current state down+peering, last acting [3,7,0,6] pg 2.74 is stuck unclean since forever, current state down+peering, last acting [3,7,0,6] pg 3.73 is stuck unclean since forever, current state down+peering, last acting [3,7,0,6] pg 2.74 is down+peering, acting [3,7,0,6] pg 3.73 is down+peering, acting [3,7,0,6] 2014-08-26 22:36:12.641585 7f5b38e507a0 0 ceph version 0.67.10 (9d446bd416c52cd785ccf048ca67737ceafcdd7f), process ceph-osd, pid 10281 2014-08-26 22:36:12.717100 7f5b38e507a0 0 filestore(/ceph/osd.1) mount FIEMAP ioctl is supported and appears to work 2014-08-26 22:36:12.717121 7f5b38e507a0 0 filestore(/ceph/osd.1) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-08-26 22:36:12.717434 7f5b38e507a0 0 filestore(/ceph/osd.1) mount detected btrfs 2014-08-26 22:36:12.717471 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs CLONE_RANGE ioctl is supported 2014-08-26 22:36:12.765009 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs SNAP_CREATE is supported 2014-08-26 22:36:12.765335 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs SNAP_DESTROY is supported 2014-08-26 22:36:12.765541 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs START_SYNC is supported (transid 3118) 2014-08-26 22:36:12.789600 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs WAIT_SYNC is supported 2014-08-26 22:36:12.808287 7f5b38e507a0 0 filestore(/ceph/osd.1) mount btrfs SNAP_CREATE_V2 is supported 2014-08-26 22:36:12.834144 7f5b38e507a0 0 filestore(/ceph/osd.1) mount syscall(SYS_syncfs, fd) fully supported 2014-08-26 22:36:12.834377 7f5b38e507a0 0 filestore(/ceph/osd.1) mount found snaps <6009082,6009083> 2014-08-26 22:36:12.834427 7f5b38e507a0 -1 filestore(/ceph/osd.1) FileStore::mount: error removing old current subvol: (22) Invalid argument 2014-08-26 22:36:12.861045 7f5b38e507a0 -1 filestore(/ceph/osd.1) mount initial op seq is 0; something is wrong 2014-08-26 22:36:12.861428 7f5b38e507a0 -1 ^[[0;31m ** ERROR: error converting store /ceph/osd.1: (22) Invalid argument^[[0m
 |