Have you re-formatted and re-added all of the lost OSDs? I've found that if you lose an OSD, you can tell Ceph the data is gone (ceph osd lost <OSDID>), but it won't believe you until it can talk to that OSDID again. If you have OSDs that are offline, you can verify that Ceph is waiting on them with ceph pg <PGID> query, looking at the recovery_state section. You're looking for probing_osds and down_osds_we_would_probe. If down_osds_we_would_probe isn't empty, then Ceph won't do anything until it's can talk to those down OSDs again. Once Ceph is able to probe all of the OSDs, you'll need to tell it to create PGs that have lost all copies. ceph pg <PGID> mark_unfound_lost revert is used when the latest version of an object was lost, but previous versions are still available. ceph pg force_create_pg <PGID> is used when all copies of a PG have been deleted. You may need to scrub the PGs too. I've been through this once, and I ran these commands many times before Ceph finally agreed to re-create the missing PGs. What your data looks like once you get the cluster healthy again, I can't say. I don't know how RDB or RadosGW will handle it. Based on the number of pools you have, I'm guessing you're using RadosGW. If so, once this is healthy, I would try to retrieve a copy of every object that RadosGW claims to have, and verify it's MD5 hash. In my case, after finally getting the PGs created, I ended up deleting the whole pool, and re-running replication. On Wed, Aug 6, 2014 at 11:33 AM, Sean Sullivan <lookcrabs at gmail.com> wrote: > I think I have a split issue or I can't seem to get rid of these objects. > How can I tell ceph to forget the objects and revert? > > How this happened is that due to the python 2.7.8/ceph bug ( a whole rack > of ceph went town (it had ubuntu 14.10 and that seemed to have 2.7.8 before > 14.04). I didn't know what was going on and tried re-installing which > killed the vast majority of the data. 2/3. The drives are gone and the data > on them is lost now. > > I tried deleting them via rados but that didn't seem to work either and > just froze there. Any help would be much appreciated. > > > Pastebin data below > http://pastebin.com/HU8yZ1ae > > > cephuser at host:~/CephPDC$ ceph --version > ceph version 0.82-524-gbf04897 (bf048976f50bd0142f291414ea893ef0f205b51a) > > cephuser at host:~/CephPDC$ ceph -s > cluster 9e0a4a8e-91fa-4643-887a-c7464aa3fd14 > health HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests > are blocked > 32 sec; recovery 478/15386946 objects degraded (0.003%); > 23/5128982 unfound (0.000%) > monmap e9: 5 mons at {kg37-12= > 10.16.0.124:6789/0,kg37-17=10.16.0.129:6789/0,kg37-23=10.16.0.135:6789/0,kg37-28=10.16.0.140:6789/0,kg37-5=10.16.0.117:6789/0}, > election epoch 1450, quorum 0,1,2,3,4 kg37-5,kg37-12,kg37-17,kg37-23,kg37-28 > mdsmap e100: 1/1/1 up {0=kg37-5=up:active} > osdmap e46061: 245 osds: 245 up, 245 in > pgmap v3268915: 22560 pgs, 19 pools, 20020 GB data, 5008 kobjects > 61956 GB used, 830 TB / 890 TB avail > 478/15386946 objects degraded (0.003%); 23/5128982 unfound > (0.000%) > 22558 active+clean > 2 active+recovering > client io 95939 kB/s rd, 80854 B/s wr, 795 op/s > > > cephuser at host:~/CephPDC$ ceph health detail > HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests are blocked > > 32 sec; 1 osds have slow requests; recovery 478/15386946 objects degraded > (0.003%); 23/5128982 unfound (0.000%) > pg 5.f4f is stuck unclean since forever, current state active+recovering, > last acting [279,115,78] > pg 5.27f is stuck unclean since forever, current state active+recovering, > last acting [213,0,258] > pg 5.f4f is active+recovering, acting [279,115,78], 10 unfound > pg 5.27f is active+recovering, acting [213,0,258], 13 unfound > 5 ops are blocked > 67108.9 sec > 5 ops are blocked > 67108.9 sec on osd.279 > 1 osds have slow requests > recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound > (0.000%) > > cephuser at host:~/CephPDC$ ceph pg 5.f4f mark_unfound_lost revert > 2014-08-06 12:59:42.282672 7f7d4a6fb700 0 -- 10.16.0.117:0/1005129 >> > 10.16.64.29:6844/718 pipe(0x7f7d4005c120 sd=4 :0 s=1 pgs=0 cs=0 l=1 > c=0x7f7d4005c3b0).fault > 2014-08-06 12:59:51.890574 7f7d4a4f9700 0 -- 10.16.0.117:0/1005129 >> > 10.16.64.29:6806/7875 pipe(0x7f7d4005f180 sd=4 :0 s=1 pgs=0 cs=0 l=1 > c=0x7f7d4005fae0).fault > pg has no unfound objects > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140807/f298a305/attachment.htm>
 |