On Wed, Oct 3, 2018 at 10:18 AM Graham Allan <gta@xxxxxxx> wrote:
Following on from my previous adventure with recovering pgs in the face
of failed OSDs, I now have my EC 4+2 pool oeprating with min_size=5
which is as things should be.
However I have one pg which is stuck in state remapped+incomplete
because it has only 4 out of 6 osds running, and I have been unable to
bring the missing two back into service.
> PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg incomplete
> pg 70.82d is remapped+incomplete, acting [2147483647,2147483647,190,448,61,315] (reducing pool .rgw.buckets.ec42 min_size from 5 may help; search ceph.com/docs for 'incomplete')
I don't think I want to do anything with min_size as that would make all
other pgs vulnerable to running dangerously undersized (unless there is
any way to force that state for only a single pg). It seems to me that
with 4/6 osds available, it should maybe be possible to force ceph to
select one or two new osds to rebalance this pg to?
I think unfortunately the easiest thing for you to fix this will be to set the min_size back to 4 until the PG is recovered (or at least has 5 shards done). This will be fixed in a later version of Ceph and probably backported, but sadly it's not done yet.
-Greg
ceph pg query gives me (snippet):
> "down_osds_we_would_probe": [
> 98,
> 233,
> 238,
> 239
> ],
> "peering_blocked_by": [],
> "peering_blocked_by_detail": [
> {
> "detail": "peering_blocked_by_history_les_bound"
> }
> ]
Of these, osd 98 appears to have a corrupt xfs filesystem
osd 239 was the original osd to hold a shard of this pg but would not
keep running, exiting with:
> /build/ceph-12.2.7/src/osd/ECBackend.cc: 619: FAILED assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset( after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))
osds 233 and 238 were otherwise evacuated (weight 0) osds to which I
imported the pg shard from osd 239 (using ceph-objectstore-tool). After
which they crash with the same assert. More specifically they seem to
crash in the same way each time the pg becomes active and starts to
backfill, on the same object:
> -9> 2018-10-03 11:30:28.174586 7f94ce9c4700 5 osd.233 pg_epoch: 704441 pg[70.82ds1( v 704329'703106 (586066'698574,704329'703106] local-lis/les=704439/704440 n=102585 ec=21494/21494 lis/c 704439/588565 les/c/f 704440/588566/0 68066
> 6/704439/704439) [820,761,105,789,562,485]/[2147483647,233,190,448,61,315]p233(1) r=1 lpr=704439 pi=[21494,704439)/4 rops=1 bft=105(2),485(5),562(4),761(1),789(3),820(0) crt=704329'703106 lcod 0'0 mlcod 0'0 active+undersized+remapped+ba
> ckfilling] backfill_pos is 70:b415ca14:::default.630943.7__shadow_Barley_GC_Project%2fBarley_GC_Project%2fRawdata%2fReads%2fCZOA.6150.7.38741.TGCTGG.fastq.gz.2~Vn8g0rMwpVY8eaW83TDzJ2mczLXAl3z.3_24:head
> -8> 2018-10-03 11:30:28.174887 7f94ce9c4700 1 -- 10.31.0.1:6854/2210291 --> 10.31.0.1:6854/2210291 -- MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2 -- 0x7f9500472280 con 0
> -7> 2018-10-03 11:30:28.174902 7f94db9de700 1 -- 10.31.0.1:6854/2210291 <== osd.233 10.31.0.1:6854/2210291 0 ==== MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2 ==== 0+0+0 (0 0 0) 0x7f9500472280
> con 0x7f94fb72b000
> -6> 2018-10-03 11:30:28.176267 7f94ead66700 5 -- 10.31.0.1:6854/2210291 >> 10.31.0.4:6880/2181727 conn(0x7f94ff2a6000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=946 cs=1 l=0). rx osd.61 seq 9 0x7f9500472500 MOSDECSubOpRe
> adReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2
> -5> 2018-10-03 11:30:28.176281 7f94ead66700 1 -- 10.31.0.1:6854/2210291 <== osd.61 10.31.0.4:6880/2181727 9 ==== MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2 ==== 786745+0+0 (875698380 0 0) 0x
> 7f9500472500 con 0x7f94ff2a6000
> -4> 2018-10-03 11:30:28.177723 7f94ead66700 5 -- 10.31.0.1:6854/2210291 >> 10.31.0.9:6920/13427 conn(0x7f94ff2bc800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=46152 cs=1 l=0). rx osd.448 seq 8 0x7f94fe9d5980 MOSDECSubOpR
> eadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2
> -3> 2018-10-03 11:30:28.177738 7f94ead66700 1 -- 10.31.0.1:6854/2210291 <== osd.448 10.31.0.9:6920/13427 8 ==== MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2 ==== 786745+0+0 (2772477454 0 0) 0x
> 7f94fe9d5980 con 0x7f94ff2bc800
> -2> 2018-10-03 11:30:28.185788 7f94ea565700 5 -- 10.31.0.1:6854/2210291 >> 10.31.0.7:6868/2012671 conn(0x7f94ff5c3800 :6854 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4193 cs=1 l=0). rx osd.190 seq 10 0x7f9500472780 MOSDECSu
> bOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2
> -1> 2018-10-03 11:30:28.185815 7f94ea565700 1 -- 10.31.0.1:6854/2210291 <== osd.190 10.31.0.7:6868/2012671 10 ==== MOSDECSubOpReadReply(70.82ds1 704441/704439 ECSubReadReply(tid=1, attrs_read=0)) v2 ==== 786745+0+0 (2670842780 0 0)
> 0x7f9500472780 con 0x7f94ff5c3800
> 0> 2018-10-03 11:30:28.194795 7f94ce9c4700 -1 /build/ceph-12.2.7/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7f94ce9c4700 time 2018-10-03 11:30:28.19026
> 0
> /build/ceph-12.2.7/src/osd/ECBackend.cc: 619: FAILED assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset( after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))
Is there anything I can do to help one of these osds (probably 233 or
238) start, such as "ceph-objectstore-tool --op repair"...? There seems
little to lose by trying but there isn't a lot of documentation on the
operations available in ceph-objectstore-tool.
I also know of the option "osd_find_best_info_ignore_history_les" but
little of what it actually does, other than being "dangerous". There are
many past intervals listed by pg query, but no "maybe_went_rw" flags so
perhaps it is safe to revert...?
--
Graham Allan
Minnesota Supercomputing Institute - gta@xxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
