ceph-objectstore-tool --data-path /var/lib/ceph/osd/cephpa1-158/
--op export-remove --pgid 85.12s0 > osd.158-85.12s0.bin
Please do that, then try to start osd.158, and report back here.
Two more questions below...
>
> 85.11 39501 0 0 0 165479411712 0
> 0 3000 stale+active+clean 3d 606021'532631
> 617659:1827554
> [124,157,68,72,102]p124
> [124,157,68,72,102]p124 2022-03-28 07:21:00.566032 2022-03-28
> 07:21:00.566032
> 85.12 39704 39704 158816 0 166350008320 0
> 0 3028 active+undersized+degraded+remapped 3d 606021'573200
> 620336:1839924
> [2147483647,2147483647,2147483647,2147483647,2147483647]p-1
> [67,91,82,2147483647,112]p67 2022-03-15 03:25:28.478280
> 2022-03-12 19:10:45.866650
> 85.25 39402 0 0 0 165108592640 0
> 0 3098 stale+down+remapped 3d 606021'521273
> 618930:1734492
> [2147483647,2147483647,2147483647,2147483647,2147483647]p-1
> [2147483647,2147483647,96,2147483647,2147483647]p96 2022-03-15
> 04:08:42.561720 2022-03-09 17:05:34.205121
> 85.33 39319 0 0 0 164740796416 0
> 0 3000 stale+active+clean 3d 606021'513259
> 617659:2125167
> [174,112,85,102,124]p174
> [174,112,85,102,124]p174 2022-03-28 07:21:12.097873 2022-03-28
> 07:21:12.097873
>
> So 85.11 and 85.33 do not look bad, after all: why are the relevant OSDs
> complaining? Is there a way to force them (OSDs) to forget about the
> chunks they possess, as apparently those have already safely migrated
> elsewhere?
>
> Indeed 85.12 is not really healthy...
> As for chunks of 85.12 and 85.25, the 3 down OSDs have:
> osd.121
> 85.12s3
> 85.25s3
> osd.158
> 85.12s0
> osd.145
> none
> I guess I can safely purge osd.145 and re-create it, then.
No!!! It contains crucial data for *other* PGs!
>
>
> As for the history of the pool, this is an EC pool with metadata in a
> SSD-backed replicated pool. At some point I realized I had made a
> mistake in the allocation rule for the "data" part, so I changed the
> relevant rule to:
>
> ~]$ ceph --cluster cephpa1 osd lspools | grep 85
> 85 csd-dataonly-ec-pool
> ~]$ ceph --cluster cephpa1 osd pool get csd-dataonly-ec-pool crush_rule
> crush_rule: csd-data-pool
>
> rule csd-data-pool {
> id 5
> type erasure
> min_size 3
> max_size 5
> step set_chooseleaf_tries 5
> step set_choose_tries 100
> step take default class big
> step choose indep 0 type host <--- this was "osd", before
> step emit
> }
Can you please share the output of `ceph osd tree` ?
We need to understand why crush is not working any more for your pool.
>
> At the time I changed the rule, there was no 'down' PG, all PGs in the
> cluster were 'active' plus possibly some other state (remapped,
> degraded, whatever) as I had added some new disk servers few days before.
Never make crush rule changes when any PG is degraded, remapped, or whatever!
They must all be active+clean to consider big changes like injecting a
new crush rule!!
Cheers, Dan
> The rule change, of course, caused some data movement and after a while
> I found those three OSDs down.
>
> Thanks!
>
> Fulvio
>
>
> On 3/30/22 16:48, Dan van der Ster wrote:
> > Hi Fulvio,
> >
> > I'm not sure why that PG doesn't register.
> > But let's look into your log. The relevant lines are:
> >
> > -635> 2022-03-30 14:49:57.810 7ff904970700 -1 log_channel(cluster)
> > log [ERR] : 85.12s0 past_intervals [616435,616454) start interval does
> > not contain the required bound [605868,616454) start
> >
> > -628> 2022-03-30 14:49:57.810 7ff904970700 -1 osd.158 pg_epoch:
> > 616454 pg[85.12s0( empty local-lis/les=0/0 n=0 ec=616435/616435 lis/c
> > 605866/605866 les/c/f 605867/605868/0 616453/616454/616454)
> > [158,168,64,102,156]/[67,91,82,121,112]p67(0) r=-1 lpr=616454
> > pi=[616435,616454)/0 crt=0'0 remapped NOTIFY mbc={}] 85.12s0
> > past_intervals [616435,616454) start interval does not contain the
> > required bound [605868,616454) start
> >
> > -355> 2022-03-30 14:49:57.816 7ff904970700 -1
> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc:
> > In function 'void PG::check_past_interval_bounds() const' thread
> > 7ff904970700 time 2022-03-30 14:49:57.811165
> >
> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc:
> > 956: ceph_abort_msg("past_interval start interval mismatch")
> >
> >
> > What is the output of `ceph pg 85.12 query` ?
> >
> > What's the history of that PG? was it moved around recently prior to this crash?
> > Are the other down osds also hosting broken parts of PG 85.12 ?
> >
> > Cheers, Dan
> >
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
