Here is the output with all OSD up and running. ceph -s: https://pastebin.com/5tMf12Lm ceph health detail: https://pastebin.com/avDhcJt0 ceph osd tree: https://pastebin.com/XEB0eUbk ceph osd pool ls detail: https://pastebin.com/ShSdmM5a On Mon, Aug 17, 2020 at 9:38 AM Martin Palma <martin@xxxxxxxx> wrote: > > Hi Peter, > > On the weekend another host was down due to power problems, which was > restarted. Therefore these outputs also include some "Degraded data > redundancy" messages. And one OSD crashed due to a disk error. > > ceph -s: https://pastebin.com/Tm8QHp52 > ceph health detail: https://pastebin.com/SrA7Bivj > ceph osd tree: https://pastebin.com/nBK8Uafd > ceph osd pool ls detail: https://pastebin.com/kYyCb7B2 > > No it's not a EC pool which has the inactive+incomplete PGs. > > ceph osd crush dump | jq '[.rules, .tunables]': https://pastebin.com/gqDTjfat > > Best, > Martin > > On Sun, Aug 16, 2020 at 3:44 PM Peter Maloney > <peter.maloney@xxxxxxxxxxxxxxxxxxxx> wrote: > > > > Dear Martin, > > > > Can you provide some details? > > > > ceph -s > > ceph health detail > > ceph osd tree > > ceph osd pool ls detail > > > > If it's EC (you implied it's not) also show the crush rules...and may as well include tunables (because greatly raising choose_total_tries, eg. 200 may be the solution to your problem): > > ceph osd crush dump | jq '[.rules, .tunables]' > > > > Peter > > > > On 8/16/20 1:18 AM, Martin Palma wrote: > > > Yes, but that didn’t help. After some time they have blocked requests again > > > and remain inactive and incomplete. > > > > > > On Sat, 15 Aug 2020 at 16:58, <ceph@xxxxxxxxxx> wrote: > > > > > >> Did you tried to restart the sayed osds? > > >> > > >> > > >> > > >> Hth > > >> > > >> Mehmet > > >> > > >> > > >> > > >> Am 12. August 2020 21:07:55 MESZ schrieb Martin Palma <martin@xxxxxxxx>: > > >> > > >>>> Are the OSDs online? Or do they refuse to boot? > > >>> Yes. They are up and running and not marked as down or out of the > > >>> cluster. > > >>>> Can you list the data with ceph-objectstore-tool on these OSDs? > > >>> If you mean the "list" operation on the PG works if an output for > > >>> example: > > >>> $ ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-63 --pgid > > >>> 22.11a --op list > > >> > > >>> ["22.11a",{"oid":"1001c1ee04f.00000007","key":"","snapid":-2,"hash":3825189146,"max":0,"pool":22,"namespace":"","max":0}] > > >> > > >>> ["22.11a",{"oid":"1000448667f.00000000","key":"","snapid":-2,"hash":4294951194,"max":0,"pool":22,"namespace":"","max":0}] > > >>> ... > > >>> If I run "ceph pg ls incomplete" in the output only one PG has > > >>> objects... all others have 0 objects. > > >>> _______________________________________________ > > >>> ceph-users mailing list -- ceph-users@xxxxxxx > > >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >> _______________________________________________ > > >> > > >> ceph-users mailing list -- ceph-users@xxxxxxx > > >> > > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > >> > > >> > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@xxxxxxx > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > > -- > > -------------------------------------------- > > Peter Maloney > > Brockmann Consult GmbH > > www.brockmann-consult.de > > Chrysanderstr. 1 > > D-21029 Hamburg, Germany > > Tel: +49 (0)40 69 63 89 - 320 > > E-mail: peter.maloney@xxxxxxxxxxxxxxxxxxxx > > Amtsgericht Hamburg HRB 157689 > > Geschäftsführer Dr. Carsten Brockmann > > -------------------------------------------- > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |