On Thu, Jun 2, 2016 at 9:07 AM, Brandon Morris, PMP <brandon.morris.pmp@xxxxxxxxx> wrote: > The only way that I was able to get back to Health_OK was to export/import. ***** Please note, any time you use the ceph_objectstore_tool you risk data loss if not done carefully. Never remove a PG until you have a known good export ***** > > Here are the steps I used: > > 1. set NOOUT, NO BACKFILL > 2. Stop the OSD's that have the erroring PG > 3. Flush the journal and export the primary version of the PG. This took 1 minute on a well-behaved PG and 4 hours on the misbehaving PG > i.e. ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-16 --journal-path /var/lib/ceph/osd/ceph-16/journal --pgid 32.10c --op export --file /root/32.10c.b.export > > 4. Import the PG into a New / Temporary OSD that is also offline, > i.e. ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-100 --journal-path /var/lib/ceph/osd/ceph-100/journal --pgid 32.10c --op export --file /root/32.10c.b.export This should be an import op and presumably to a different data path and journal path more like the following? ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-101 --journal-path /var/lib/ceph/osd/ceph-101/journal --pgid 32.10c --op import --file /root/32.10c.b.export Just trying to clarify for anyone that comes across this thread in the future. Cheers, Brad > > 5. remove the PG from all other OSD's (16, 143, 214, and 448 in your case it looks like) > 6. Start cluster OSD's > 7. Start the temporary OSD's and ensure 32.10c backfills correctly to the 3 OSD's it is supposed to be on. > > This is similar to the recovery process described in this post from 04/09/2015: http://ceph-users.ceph.narkive.com/lwDkR2fZ/recovering-incomplete-pgs-with-ceph-objectstore-tool Hopefully it works in your case too and you can the cluster back to a state that you can make the CephFS directories smaller. > > - Brandon _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |