I don't see that same_interval_since being cleared by split.
PG::split_into() copies the history from the parent PG to child. The
only code in Luminous that I see that clears it is in
ceph_objectstore_tool.cc
David
On 10/16/17 3:59 PM, Gregory Farnum wrote:
On Mon, Oct 16, 2017 at 3:49 PM Dejan Lesjak <dejan.lesjak@xxxxxx> wrote:
On 17. okt. 2017, at 00:23, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
On Mon, Oct 16, 2017 at 8:24 AM Dejan Lesjak <dejan.lesjak@xxxxxx>
wrote:
On 10/16/2017 02:02 PM, Dejan Lesjak wrote:
Hi,
During rather high load and rebalancing, a couple of our OSDs crashed
and they fail to start. This is from the log:
-2> 2017-10-16 13:27:50.235204 7f5e4c3bae80 0 osd.1 442123
load_pgs
opened 370 pgs
-1> 2017-10-16 13:27:50.239175 7f5e4c3bae80 1 osd.1 442123
build_past_intervals_parallel over 439159-439159
0> 2017-10-16 13:27:50.261883 7f5e4c3bae80 -1
/var/tmp/portage/sys-cluster/ceph-12.2.1/work/ceph-12.2.1/src/osd/OSD.cc:
In function 'void OSD::build_past_intervals_parallel()' thread
7f5e4c3bae80 time 2017-10-16 13:27:50.260062
/var/tmp/portage/sys-cluster/ceph-12.2.1/work/ceph-12.2.1/src/osd/OSD.cc:
4177: FAILED assert(p.same_interval_since)
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e)
luminous
(stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x55e4caa18592]
2: (OSD::build_past_intervals_parallel()+0x1d7b) [0x55e4ca453e8b]
3: (OSD::load_pgs()+0x14cb) [0x55e4ca45564b]
4: (OSD::init()+0x2227) [0x55e4ca467327]
5: (main()+0x2d5a) [0x55e4ca379b1a]
6: (__libc_start_main()+0xf1) [0x7f5e48ee35d1]
7: (_start()+0x2a) [0x55e4ca4039aa]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
Does anybody know how to fix or further debug this?
Bumped logging to 10 and posted log to https://pastebin.com/raw/StTeYWRt
From "10.1fces2 needs 439159-0" it seems osd (osd.1) gets stuck at pg
10.1fce. Yet pg map doesn't show osd.1 for this pg:
# ceph pg map 10.1fce
osdmap e443665 pg 10.1fce (10.1fce) -> up [110,213,132,182] acting
[110,213,132,182]
Hmm, this is odd. What caused your rebalancing exactly? Can you turn on
the OSD with debugging set to 20, and then upload the log file using
ceph-post-file?
The specific assert you're hitting here is supposed to cope with PGs
that have been imported (via the ceph-objectstore-tool). But obviously
something has gone wrong here.
It started when we bumped the number of PGs for a pool (from 2048 to 8192).
I’ve sent the log with ID 3a6dea4f-05d7-4c15-9f7e-2d95d99195ba
It actually seems similar than http://tracker.ceph.com/issues/21142 in
that the pg found in log seems empty if checked with ceph-objectstore-tool
and removing it allows the osd to start. At least on one osd, I’ve not
tried that yet on all of the failed ones.
Ah. I bet we are default-constructing the "child" PGs from split with that
value set to zero, so it's incorrectly being flagged for later use. David,
does that make sense to you? Do you think it's reasonable to fix it by just
checking for other default-initialized values as part of that branch check?
(I note that this code got removed once Luminous branched, so hopefully
there's a simple fix we can apply!)
Dejan, did you make sure the OSD you tried that on has re-created the
removed PG and populated it with data? If so I think you ought to be fine
removing any empty PGs which are causing this assert.
-Greg
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
