Thanks for help, I have solved this problem in other way. After many time spent on importing data to osd where data was in past, always pg was incomplete. I tried to import to one osd, to all osd, always pg have status 'incomplete'. At last I do ceph pg force_create_pg 0.109 and pg was creating state, but never end. Then after a day I rebooted all servers in cluster and after systems up cluster was clean. So I repeat importing procedure because I won't lost 18GB data of exported pg. After import pg was incomplete state, reboot doesn't help. My next step is to do in the same time 'Ceph pg force_create_pg 0.109' and 'dsh -acM reboot' (reboots all my servers) After reboot starts to repair cluster, backfilling data to other osd. When process ended cluster has clean, and 18GB of exported data doesn't disappear. I think cluster have no time to delete data but sets pg as complete. I write that You don't know how to get incomplete state on PG. In my case I have full cluster, some disks have 100% used space and won't up. There was no possibility to start osd, without delete data. So I listed /var/lib/ceph/osd/ceph-7/current for id of pg, next check if this pg is active+clean and deleted pg 0.109. I think in time between check 'ceph pg dump' and delete data of /var/lib/ceph/osd/ceph-7/current/0.109_head cluster starts to remap data to other osd, and doesn't end before deleting. I don't set noout, norecover, nobackfill flags, and don't stop osd's, it's my mistake, so I lost only ca. 2GB data. You could try get incomplete pg by # check 'ceph pg dump' for osd on any pg. # stop all osd with this pg # go to one of this osd and remove some pice of data in /var/lib/ceph/osd/ceph-$id/current/$pgid_head/DIR*/DIR*/ # remove all data of this pg on other osds # start all osd's - You probably have incomplete pg Thanks for Your work, it could help me don't waste my time. Regards, Mateusz -----Original Message----- From: Mykola Golub [mailto:mgolub@xxxxxxxxxxxx] Sent: Friday, June 19, 2015 2:05 PM To: Mateusz Skała Cc: ceph-users@xxxxxxxxxxxxxx; Samuel Just; Sage Weil Subject: Re: incomplete pg, recovery some data On Thu, Jun 18, 2015 at 01:24:38PM +0200, Mateusz Skała wrote: > Hi, > > After some hardware errors one of pg in our backup server is 'incomplete'. > > I do export pg without problems like here: > https://ceph.com/community/incomplete-pgs-oh-my/ > > After remove pg from all osd's and import pg to one of osd pg is > still 'incomplete'. > > I want to recover only some pice of data from this rbd so if I lost > something then nothing happened. How can I tell ceph to accept this pg > as complete and clean? I have a patch for ceph-objectstore-tool, which adds mark-complete operation, as it has been suggested by Sam in http://tracker.ceph.com/issues/10098 https://github.com/ceph/ceph/pull/5031 It has not been reviewed yet and not tested well though because I don't know a simple way how to get an incomplete pg. You might want to try it on your own risk. -- Mykola Golub _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |