Ciao Dan,
thanks for your time!
So you are suggesting that my problems with PG 85.25 may somehow resolve
if I manage to bring up the three OSDs currently "down" (possibly due to
PG 85.12, and other PGs)?
Looking for the string 'start interval does not contain the required
bound' I found similar errors in the three OSDs:
osd.158: 85.12s0
osd.145: 85.33s0
osd.121: 85.11s0
Here is the output of "pg 85.12 query":
https://pastebin.ubuntu.com/p/ww3JdwDXVd/
and its status (also showing the other 85.XX, for reference):
85.11 39501 0 0 0 165479411712 0
0 3000 stale+active+clean 3d 606021'532631
617659:1827554
[124,157,68,72,102]p124
[124,157,68,72,102]p124 2022-03-28 07:21:00.566032 2022-03-28
07:21:00.566032
85.12 39704 39704 158816 0 166350008320 0
0 3028 active+undersized+degraded+remapped 3d 606021'573200
620336:1839924
[2147483647,2147483647,2147483647,2147483647,2147483647]p-1
[67,91,82,2147483647,112]p67 2022-03-15 03:25:28.478280
2022-03-12 19:10:45.866650
85.25 39402 0 0 0 165108592640 0
0 3098 stale+down+remapped 3d 606021'521273
618930:1734492
[2147483647,2147483647,2147483647,2147483647,2147483647]p-1
[2147483647,2147483647,96,2147483647,2147483647]p96 2022-03-15
04:08:42.561720 2022-03-09 17:05:34.205121
85.33 39319 0 0 0 164740796416 0
0 3000 stale+active+clean 3d 606021'513259
617659:2125167
[174,112,85,102,124]p174
[174,112,85,102,124]p174 2022-03-28 07:21:12.097873 2022-03-28
07:21:12.097873
So 85.11 and 85.33 do not look bad, after all: why are the relevant OSDs
complaining? Is there a way to force them (OSDs) to forget about the
chunks they possess, as apparently those have already safely migrated
elsewhere?
Indeed 85.12 is not really healthy...
As for chunks of 85.12 and 85.25, the 3 down OSDs have:
osd.121
85.12s3
85.25s3
osd.158
85.12s0
osd.145
none
I guess I can safely purge osd.145 and re-create it, then.
As for the history of the pool, this is an EC pool with metadata in a
SSD-backed replicated pool. At some point I realized I had made a
mistake in the allocation rule for the "data" part, so I changed the
relevant rule to:
~]$ ceph --cluster cephpa1 osd lspools | grep 85
85 csd-dataonly-ec-pool
~]$ ceph --cluster cephpa1 osd pool get csd-dataonly-ec-pool crush_rule
crush_rule: csd-data-pool
rule csd-data-pool {
id 5
type erasure
min_size 3
max_size 5
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default class big
step choose indep 0 type host <--- this was "osd", before
step emit
}
At the time I changed the rule, there was no 'down' PG, all PGs in the
cluster were 'active' plus possibly some other state (remapped,
degraded, whatever) as I had added some new disk servers few days before.
The rule change, of course, caused some data movement and after a while
I found those three OSDs down.
Thanks!
Fulvio
On 3/30/22 16:48, Dan van der Ster wrote:
Hi Fulvio,
I'm not sure why that PG doesn't register.
But let's look into your log. The relevant lines are:
-635> 2022-03-30 14:49:57.810 7ff904970700 -1 log_channel(cluster)
log [ERR] : 85.12s0 past_intervals [616435,616454) start interval does
not contain the required bound [605868,616454) start
-628> 2022-03-30 14:49:57.810 7ff904970700 -1 osd.158 pg_epoch:
616454 pg[85.12s0( empty local-lis/les=0/0 n=0 ec=616435/616435 lis/c
605866/605866 les/c/f 605867/605868/0 616453/616454/616454)
[158,168,64,102,156]/[67,91,82,121,112]p67(0) r=-1 lpr=616454
pi=[616435,616454)/0 crt=0'0 remapped NOTIFY mbc={}] 85.12s0
past_intervals [616435,616454) start interval does not contain the
required bound [605868,616454) start
-355> 2022-03-30 14:49:57.816 7ff904970700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc:
In function 'void PG::check_past_interval_bounds() const' thread
7ff904970700 time 2022-03-30 14:49:57.811165
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.22/rpm/el7/BUILD/ceph-14.2.22/src/osd/PG.cc:
956: ceph_abort_msg("past_interval start interval mismatch")
What is the output of `ceph pg 85.12 query` ?
What's the history of that PG? was it moved around recently prior to this crash?
Are the other down osds also hosting broken parts of PG 85.12 ?
Cheers, Dan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
