I think I have a split issue or I can't seem to get rid of these objects. How can I tell ceph to forget the objects and revert? How this happened is that due to the python 2.7.8/ceph bug ( a whole rack of ceph went town (it had ubuntu 14.10 and that seemed to have 2.7.8 before 14.04). I didn't know what was going on and tried re-installing which killed the vast majority of the data. 2/3. The drives are gone and the data on them is lost now. I tried deleting them via rados but that didn't seem to work either and just froze there. Any help would be much appreciated. Pastebin data below http://pastebin.com/HU8yZ1ae cephuser at host:~/CephPDC$ ceph --version ceph version 0.82-524-gbf04897 (bf048976f50bd0142f291414ea893ef0f205b51a) cephuser at host:~/CephPDC$ ceph -s cluster 9e0a4a8e-91fa-4643-887a-c7464aa3fd14 health HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests are blocked > 32 sec; recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound (0.000%) monmap e9: 5 mons at {kg37-12= 10.16.0.124:6789/0,kg37-17=10.16.0.129:6789/0,kg37-23=10.16.0.135:6789/0,kg37-28=10.16.0.140:6789/0,kg37-5=10.16.0.117:6789/0}, election epoch 1450, quorum 0,1,2,3,4 kg37-5,kg37-12,kg37-17,kg37-23,kg37-28 mdsmap e100: 1/1/1 up {0=kg37-5=up:active} osdmap e46061: 245 osds: 245 up, 245 in pgmap v3268915: 22560 pgs, 19 pools, 20020 GB data, 5008 kobjects 61956 GB used, 830 TB / 890 TB avail 478/15386946 objects degraded (0.003%); 23/5128982 unfound (0.000%) 22558 active+clean 2 active+recovering client io 95939 kB/s rd, 80854 B/s wr, 795 op/s cephuser at host:~/CephPDC$ ceph health detail HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests are blocked > 32 sec; 1 osds have slow requests; recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound (0.000%) pg 5.f4f is stuck unclean since forever, current state active+recovering, last acting [279,115,78] pg 5.27f is stuck unclean since forever, current state active+recovering, last acting [213,0,258] pg 5.f4f is active+recovering, acting [279,115,78], 10 unfound pg 5.27f is active+recovering, acting [213,0,258], 13 unfound 5 ops are blocked > 67108.9 sec 5 ops are blocked > 67108.9 sec on osd.279 1 osds have slow requests recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound (0.000%) cephuser at host:~/CephPDC$ ceph pg 5.f4f mark_unfound_lost revert 2014-08-06 12:59:42.282672 7f7d4a6fb700 0 -- 10.16.0.117:0/1005129 >> 10.16.64.29:6844/718 pipe(0x7f7d4005c120 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7d4005c3b0).fault 2014-08-06 12:59:51.890574 7f7d4a4f9700 0 -- 10.16.0.117:0/1005129 >> 10.16.64.29:6806/7875 pipe(0x7f7d4005f180 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f7d4005fae0).fault pg has no unfound objects -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140806/d8ed4271/attachment.htm>
 |