Glad to hear it helped.
This particular option is ultra dangerous, so imho its obfuscated name is just perfect!
Finally, since I didn't mention it earlier, don't forget to disable the option and restart the relevant OSDs now that their active again. And it would be sensible to deep scrub that PG now.
Cheers,
Dan
On Wed, Dec 19, 2018, 5:46 PM Fulvio Galeazzi <fulvio.galeazzi@xxxxxxx wrote:
Ciao Dan,
thanks a lot for your message! :-)
Indeed, the procedure you outlined did the trick and I am now back to
healthy state.
--yes-i-really-really-love-ceph-parameter-names !!!
Ciao ciao
Fulvio
-------- Original Message --------
Subject: Re: Luminous (12.2.8 on CentOS), recover or
recreate incomplete PG
From: Dan van der Ster <dan@xxxxxxxxxxxxxx>
To: fulvio.galeazzi@xxxxxxx
CC: ceph-users <ceph-users@xxxxxxxxxxxxxx>
Date: 12/18/2018 11:38 AM
> Hi Fulvio!
>
> Are you able to query that pg -- which osd is it waiting for?
>
> Also, since you're prepared for data loss anyway, you might have
> success setting osd_find_best_info_ignore_history_les=true on the
> relevant osds (set it conf, restart those osds).
>
> -- dan
>
>
> -- dan
>
> On Tue, Dec 18, 2018 at 11:31 AM Fulvio Galeazzi
> <fulvio.galeazzi@xxxxxxx> wrote:
>>
>> Hallo Cephers,
>> I am stuck with an incomplete PG and am seeking help.
>>
>> At some point I had a bad configuration for gnocchi which caused a
>> flooding of tiny objects to the backend Ceph rados pool. While cleaning
>> things up, the load on the OSD disks was such that 3 of them "commited
>> suicide" and were marked down.
>> Now that the situation is calm, I am left with one stubborn
>> incomplete PG.
>>
>> PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
>> pg 107.33 is incomplete, acting [41,22,156] (reducing pool
>> gnocchi-ct1-cl1 min_size from 2 may help; search ceph.com/docs for
>> 'incomplete')
>> (by the way, reducing min_size did not help)
>>
>> I found this page and tried to follow the procedure outlined:
>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-July/019674.html
>>
>> On one of the 3 replicas, the "PG export" produced some decently
>> sized file, but when I tried to import it on the acting OSD I got error:
>>
>> [root@r1srv07.ct1 ~]# ceph-objectstore-tool --data-path
>> /var/lib/ceph/osd/ceph-41 --op import --file /tmp/recover.107.33 --force
>> pgid 107.33 already exists
>>
>>
>> Questions now is: could anyone please suggest a recovery procedure? Note
>> that for this specific case I would not mind wiping the PG.
>>
>> Thanks for your help!
>>
>> Fulvio
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
