All inactive and incomplete PGs are blocked by OSD 81 which does not
exist anymore:
...
"down_osds_we_would_probe": [
81
],
"peering_blocked_by": [],
"peering_blocked_by_detail": [
{
"detail": "peering_blocked_by_history_les_bound"
}
]
...
Here the full output: https://pastebin.com/V5EPZ0N7
On Thu, Aug 20, 2020 at 10:58 AM Dan van der Ster <dan@xxxxxxxxxxxxxx> wrote:
>
> Something else to help debugging is
>
> ceph pg 17.173 query
>
> at the end it should say why the pg is incomplete.
>
> -- dan
>
>
>
> On Thu, Aug 20, 2020 at 10:01 AM Eugen Block <eblock@xxxxxx> wrote:
> >
> > Hi Martin,
> >
> > have you seen this blog post [1]? It describes how to recover from
> > inactive and incomplete PGs (on a size 1 pool). I haven't tried any of
> > that but it could be worth a try. Apparently it only would work if the
> > affected PGs have 0 objects but that seems to be the case, right?
> >
> > Regards,
> > Eugen
> >
> > [1]
> > https://medium.com/opsops/recovering-ceph-from-reduced-data-availability-3-pgs-inactive-3-pgs-incomplete-b97cbcb4b5a1
> >
> >
> > Zitat von Martin Palma <martin@xxxxxxxx>:
> >
> > > If Ceph consultants are reading this please feel free to contact me
> > > off list. We are seeking for someone who can help us of course we will
> > > pay.
> > >
> > >
> > >
> > > On Mon, Aug 17, 2020 at 12:50 PM Martin Palma <martin@xxxxxxxx> wrote:
> > >>
> > >> After doing some research I suspect the problem is that during the
> > >> cluster was backfilling an OSD was removed.
> > >>
> > >> Now the PGs which are inactive and incomplete have all the same
> > >> (removed OSD) in the "down_osds_we_would_probe" output and the peering
> > >> is blocked by "peering_blocked_by_history_les_bound". We tried to set
> > >> the "osd_find_best_info_ignore_history_les = true" but with no success
> > >> the OSDs keep in a peering loop.
> > >>
> > >> On Mon, Aug 17, 2020 at 9:53 AM Martin Palma <martin@xxxxxxxx> wrote:
> > >> >
> > >> > Here is the output with all OSD up and running.
> > >> >
> > >> > ceph -s: https://pastebin.com/5tMf12Lm
> > >> > ceph health detail: https://pastebin.com/avDhcJt0
> > >> > ceph osd tree: https://pastebin.com/XEB0eUbk
> > >> > ceph osd pool ls detail: https://pastebin.com/ShSdmM5a
> > >> >
> > >> > On Mon, Aug 17, 2020 at 9:38 AM Martin Palma <martin@xxxxxxxx> wrote:
> > >> > >
> > >> > > Hi Peter,
> > >> > >
> > >> > > On the weekend another host was down due to power problems, which was
> > >> > > restarted. Therefore these outputs also include some "Degraded data
> > >> > > redundancy" messages. And one OSD crashed due to a disk error.
> > >> > >
> > >> > > ceph -s: https://pastebin.com/Tm8QHp52
> > >> > > ceph health detail: https://pastebin.com/SrA7Bivj
> > >> > > ceph osd tree: https://pastebin.com/nBK8Uafd
> > >> > > ceph osd pool ls detail: https://pastebin.com/kYyCb7B2
> > >> > >
> > >> > > No it's not a EC pool which has the inactive+incomplete PGs.
> > >> > >
> > >> > > ceph osd crush dump | jq '[.rules, .tunables]':
> > >> https://pastebin.com/gqDTjfat
> > >> > >
> > >> > > Best,
> > >> > > Martin
> > >> > >
> > >> > > On Sun, Aug 16, 2020 at 3:44 PM Peter Maloney
> > >> > > <peter.maloney@xxxxxxxxxxxxxxxxxxxx> wrote:
> > >> > > >
> > >> > > > Dear Martin,
> > >> > > >
> > >> > > > Can you provide some details?
> > >> > > >
> > >> > > > ceph -s
> > >> > > > ceph health detail
> > >> > > > ceph osd tree
> > >> > > > ceph osd pool ls detail
> > >> > > >
> > >> > > > If it's EC (you implied it's not) also show the crush
> > >> rules...and may as well include tunables (because greatly raising
> > >> choose_total_tries, eg. 200 may be the solution to your problem):
> > >> > > > ceph osd crush dump | jq '[.rules, .tunables]'
> > >> > > >
> > >> > > > Peter
> > >> > > >
> > >> > > > On 8/16/20 1:18 AM, Martin Palma wrote:
> > >> > > > > Yes, but that didn’t help. After some time they have
> > >> blocked requests again
> > >> > > > > and remain inactive and incomplete.
> > >> > > > >
> > >> > > > > On Sat, 15 Aug 2020 at 16:58, <ceph@xxxxxxxxxx> wrote:
> > >> > > > >
> > >> > > > >> Did you tried to restart the sayed osds?
> > >> > > > >>
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> Hth
> > >> > > > >>
> > >> > > > >> Mehmet
> > >> > > > >>
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> Am 12. August 2020 21:07:55 MESZ schrieb Martin Palma
> > >> <martin@xxxxxxxx>:
> > >> > > > >>
> > >> > > > >>>> Are the OSDs online? Or do they refuse to boot?
> > >> > > > >>> Yes. They are up and running and not marked as down or out of the
> > >> > > > >>> cluster.
> > >> > > > >>>> Can you list the data with ceph-objectstore-tool on these OSDs?
> > >> > > > >>> If you mean the "list" operation on the PG works if an output for
> > >> > > > >>> example:
> > >> > > > >>> $ ceph-objectstore-tool --data-path
> > >> /var/lib/ceph/osd/ceph-63 --pgid
> > >> > > > >>> 22.11a --op list
> > >> > > > >>
> > >> > > > >>>
> > >> ["22.11a",{"oid":"1001c1ee04f.00000007","key":"","snapid":-2,"hash":3825189146,"max":0,"pool":22,"namespace":"","max":0}]
> > >> > > > >>
> > >> > > > >>>
> > >> ["22.11a",{"oid":"1000448667f.00000000","key":"","snapid":-2,"hash":4294951194,"max":0,"pool":22,"namespace":"","max":0}]
> > >> > > > >>> ...
> > >> > > > >>> If I run "ceph pg ls incomplete" in the output only one PG has
> > >> > > > >>> objects... all others have 0 objects.
> > >> > > > >>> _______________________________________________
> > >> > > > >>> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> > > > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> > > > >> _______________________________________________
> > >> > > > >>
> > >> > > > >> ceph-users mailing list -- ceph-users@xxxxxxx
> > >> > > > >>
> > >> > > > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> > > > >>
> > >> > > > >>
> > >> > > > > _______________________________________________
> > >> > > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > >> > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > --------------------------------------------
> > >> > > > Peter Maloney
> > >> > > > Brockmann Consult GmbH
> > >> > > > www.brockmann-consult.de
> > >> > > > Chrysanderstr. 1
> > >> > > > D-21029 Hamburg, Germany
> > >> > > > Tel: +49 (0)40 69 63 89 - 320
> > >> > > > E-mail: peter.maloney@xxxxxxxxxxxxxxxxxxxx
> > >> > > > Amtsgericht Hamburg HRB 157689
> > >> > > > Geschäftsführer Dr. Carsten Brockmann
> > >> > > > --------------------------------------------
> > >> > > >
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
