On Thu, 14 Jun 2018, Wyllys Ingersoll wrote: > I cut out a HUGE list of "purged_snaps" to keep this a little shorter... > > $ cat 1.10e.txt > { > "state": "incomplete", > "snap_trimq": "[]", > "snap_trimq_len": 0, > "epoch": 465904, > "up": [ > 52, > 23, > 20 > ], > "acting": [ > 52, > 23, > 20 > ], > "info": { > "pgid": "1.10e", > "last_update": "438490'293946", > "last_complete": "438490'293946", > "log_tail": "427182'292446", > "last_user_version": 0, > "last_backfill": "MIN", ... > "peer_info": [ > { > "peer": "5", > "pgid": "1.10e", > "last_update": "438490'293946", > "last_complete": "438490'293946", > "log_tail": "427182'292446", > "last_user_version": 0, > "last_backfill": "MIN", ... > }, > { > "peer": "10", > "pgid": "1.10e", > "last_update": "438490'293946", > "last_complete": "438490'293946", > "log_tail": "427182'292446", > "last_user_version": 0, > "last_backfill": "MIN", ... > } > ], It looks like all of the copies of this PG are in fact incomplete (partially backfilled, not the complete set of objects). You must have lost a disk somewhere? Is there another copy? If not, then as a last resort you can go look at each one, see which copy of the PG has the most objects, and mark it complete, and everything else where backfill from there. That is almost certainly going to admit defeat and lose some data, though. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html