On Mon, 6 Aug 2018, Brayan Perera wrote: > Hi Sage, > > > Above is from osdmap which I have extracted from one of the ceph-mons > > =================== > pool 111 'moc_test' replicated size 1 min_size 1 crush_rule 2 > object_hash rjenkins pg_num 16 pgp_num 16 last_change 6337 flags > hashpspool stripe_width 0 > > pg_temp 111.1 [50,33,25] > pg_temp 111.2 [50,33,40] > pg_temp 111.3 [45,40,6] > pg_temp 111.4 [45,0,33] > pg_temp 111.7 [6,32,19] > pg_temp 111.8 [22,14,58] > pg_temp 111.9 [40,9,50] > pg_temp 111.a [15,14,22] > pg_temp 111.b [50,6,22] > pg_temp 111.c [59,37,19] > pg_temp 111.d [29,0,52] > pg_temp 111.f [33,32,6] > =================== Yep, that's the bug. I've opened http://tracker.ceph.com/issues/26866 and I've pushed a temporary workaround to that same branch that should get your mons started. Builds will take an hour-ish again! Thanks- sage > > > Thanks & Regards, > Brayan > > On Mon, Aug 6, 2018 at 10:21 PM, Brayan Perera <brayan.perera@xxxxxxxxx> wrote: > > Hi Sage, > > > > > > No, this is a regular pool. > > > > This pool is a test pool. Other pools are having production data. > > > > Is there a way to get rid of this pool as a workaround ? > > > > Thanks & Regards, > > Brayan > > > > On Mon, Aug 6, 2018 at 10:20 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: > >> Is that an EC pool? > >> > >> On Mon, 6 Aug 2018, Brayan Perera wrote: > >> > >>> Hi Sage, > >>> > >>> Please find the attachment. Sorry I could not use ceph-post-file option. > >>> > >>> Looks like one of the pgs mapping is invalid. 111.4 osd.-1 > >>> > >>> Is there fix for this issue ? > >>> > >>> Really appreciate your help on this. > >>> > >>> > >>> Thanks & Regards, > >>> Brayan > >>> > >>> > >>> On Mon, Aug 6, 2018 at 8:46 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: > >>> > no problem, just wanted to make sure we weren't waiting for a build that > >>> > did exist :) > >>> > > >>> > On Mon, 6 Aug 2018, Brayan Perera wrote: > >>> > > >>> >> We are on Centos 7. Sorry I forgot to inform that. > >>> >> > >>> >> Regards, > >>> >> Brayan > >>> >> > >>> >> On Mon, Aug 6, 2018 at 8:43 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: > >>> >> > are you on centos7 or ubuntu xenial? those are the two distros being > >>> >> > built... > >>> >> > > >>> >> > On Mon, 6 Aug 2018, Brayan Perera wrote: > >>> >> > > >>> >> >> Hi Sage, > >>> >> >> > >>> >> >> Thanks for the change. Will test once build is available. Only > >>> >> >> ceph-mon rpm is enough or we have to use whole repository ? > >>> >> >> > >>> >> >> Thanks & Regards, > >>> >> >> Brayan > >>> >> >> > >>> >> >> > >>> >> >> On Mon, Aug 6, 2018 at 8:36 PM, Sage Weil <sage@xxxxxxxxxx> wrote: > >>> >> >> > Also, feel free to you 'ceph-post-file <file>' instead of an attachment > >>> >> >> > (it will restrict access to the file to ceph developers w/ access to the > >>> >> >> > shared test lab). > >>> >> >> > > >>> >> >> > sage > >>> >> >> > > >>> >> >> > > >>> >> >> > On Mon, 6 Aug 2018, Sage Weil wrote: > >>> >> >> > > >>> >> >> >> On Mon, 6 Aug 2018, Brayan Perera wrote: > >>> >> >> >> > Hi Sage, > >>> >> >> >> > > >>> >> >> >> > Please find the attachment. > >>> >> >> >> > >>> >> >> >> You need to wait until these builds complete, > >>> >> >> >> https://shaman.ceph.com/repos/ceph/wip-rmap-thing/520b28b2e461e2fdb317b1566f48ae844c08cb43/, > >>> >> >> >> then install on one of the mon hosts, and then run teh command in order to > >>> >> >> >> see the new debug output. (Or apply the patch locally.) > >>> >> >> >> > >>> >> >> >> Thanks! > >>> >> >> >> sage > >>> >> >> >> > >>> >> >> >> > > >>> >> >> >> > On Mon, Aug 6, 2018 at 7:54 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > >>> >> >> >> > > On Mon, 6 Aug 2018, Brayan Perera wrote: > >>> >> >> >> > >> Hi Sage, > >>> >> >> >> > >> > >>> >> >> >> > >> This is happened on production ceph cluster. Ceph mons were suddenly went > >>> >> >> >> > >> offline. > >>> >> >> >> > >> > >>> >> >> >> > >> When we check the logs while restarting the ceph-mon services we got above > >>> >> >> >> > >> error. Complete log entry is attached as 'ceph_mon_start_error.txt'. > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> Since issue started on one of the test pools. () > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> Pool is having size as 1. But the associated crush rule is having min size > >>> >> >> >> > >> set to 2. Can this cause the issue ? > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> As a workaround we we trying to change the crush rule and inject the crush > >>> >> >> >> > >> map. But still crush map is not getting changed. > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> /usr/bin/ceph-monstore-tool /var/lib/ceph/mon/ceph-cephmon rewrite-crush -- > >>> >> >> >> > >> --crush ./osdmap_crush.updated_compiled > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> Do we have an option to change pool size in the osdmap directly inside > >>> >> >> >> > >> ceph-mon ? > >>> >> >> >> > > > >>> >> >> >> > > I would not mess with the monitor's osdmap until we understand what is > >>> >> >> >> > > wrong. The sequence you describe shouldn't have caused a crash (and a > >>> >> >> >> > > quick attempt to reproduce this failed). > >>> >> >> >> > > > >>> >> >> >> > > I pushed a branch, https://github.com/ceph/ceph-ci/commit/wip-rmap-thing, > >>> >> >> >> > > with some debug output. This will build at shaman.ceph.com shortly. Do > >>> >> >> >> > > you mind running this on one of your mons with > >>> >> >> >> > > > >>> >> >> >> > > ceph-mon -i `hostname` -d 2>&1 | tee output > >>> >> >> >> > > > >>> >> >> >> > > and capturing the output? It looks like an out-of-bounds access on > >>> >> >> >> > > acting_rmap but it's not clear to me how that can happen. > >>> >> >> >> > > > >>> >> >> >> > > Thanks! > >>> >> >> >> > > sage > >>> >> >> >> > > > >>> >> >> >> > > > >>> >> >> >> > > > > >>> >> >> >> > >> > >>> >> >> >> > >> Thanks & Regards, > >>> >> >> >> > >> Brayan > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> On Mon, Aug 6, 2018 at 6:50 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote: > >>> >> >> >> > >> > >>> >> >> >> > >> > Moving to ceph-devel. > >>> >> >> >> > >> > > >>> >> >> >> > >> > On Mon, 6 Aug 2018, Brayan Perera wrote: > >>> >> >> >> > >> > > Dear All, > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > We have encounter an issue on ceph mons, where all the ceph-mon instances > >>> >> >> >> > >> > > went offline with following error. > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > ======================= > >>> >> >> >> > >> > > 0> 2018-08-06 18:20:56.373266 7f9e4b98d700 -1 *** Caught signal > >>> >> >> >> > >> > > (Segmentation fault) ** > >>> >> >> >> > >> > > in thread 7f9e4b98d700 thread_name:cpu_tp > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous > >>> >> >> >> > >> > > (stable) > >>> >> >> >> > >> > > 1: (()+0x8f51b1) [0x560c971ab1b1] > >>> >> >> >> > >> > > 2: (()+0xf6d0) [0x7f9e541106d0] > >>> >> >> >> > >> > > 3: (void std::vector<pg_t, > >>> >> >> >> > >> > > mempool::pool_allocator<(mempool::pool_index_t)16, pg_t> > >>> >> >> >> > >> > > >::_M_emplace_back_aux<pg_t const&>(pg_t const&)+0x69) [0x560c96fb5fd9] > >>> >> >> >> > >> > > 4: (OSDMapMapping::_build_rmap(OSDMap const&)+0x209) [0x560c96fb5389] > >>> >> >> >> > >> > > 5: (OSDMapMapping::_finish(OSDMap const&)+0x11) [0x560c96fb53c1] > >>> >> >> >> > >> > > 6: (ParallelPGMapper::Job::finish_one()+0x82) [0x560c96fb4332] > >>> >> >> >> > >> > > 7: (ParallelPGMapper::WQ::_process(ParallelPGMapper::Item*, > >>> >> >> >> > >> > > ThreadPool::TPHandle&)+0x7f) [0x560c96fb43ff] > >>> >> >> >> > >> > > 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa8e) [0x560c96eaf59e] > >>> >> >> >> > >> > > 9: (ThreadPool::WorkThread::entry()+0x10) [0x560c96eb0480] > >>> >> >> >> > >> > > 10: (()+0x7e25) [0x7f9e54108e25] > >>> >> >> >> > >> > > 11: (clone()+0x6d) [0x7f9e512febad] > >>> >> >> >> > >> > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed > >>> >> >> >> > >> > > to interpret this. > >>> >> >> >> > >> > > ============================ > >>> >> >> >> > >> > > >>> >> >> >> > >> > I haven't seen this one before. I take it it happens reliably when you > >>> >> >> >> > >> > try to start the OSD? Can you reproduce the bug and generate a mon log > >>> >> >> >> > >> > file by setting 'debug mon = 20'? > >>> >> >> >> > >> > > >>> >> >> >> > >> > Thanks! > >>> >> >> >> > >> > sage > >>> >> >> >> > >> > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > Before this we have change one of the test pools size to 1 from 3 for > >>> >> >> >> > >> > > testing purpose. > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > Is there a way to recover this without loosing the data ? > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > Thanks in advance. > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > Regards, > >>> >> >> >> > >> > > Brayan > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > -- > >>> >> >> >> > >> > > Best Regards, > >>> >> >> >> > >> > > Brayan Perera > >>> >> >> >> > >> > > > >>> >> >> >> > >> > > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> > >>> >> >> >> > >> -- > >>> >> >> >> > >> Best Regards, > >>> >> >> >> > >> Brayan Perera > >>> >> >> >> > >> > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > -- > >>> >> >> >> > Best Regards, > >>> >> >> >> > Brayan Perera > >>> >> >> >> > > >>> >> >> >> -- > >>> >> >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >>> >> >> >> the body of a message to majordomo@xxxxxxxxxxxxxxx > >>> >> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> >> >> >> > >>> >> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Best Regards, > >>> >> >> Brayan Perera > >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Best Regards, > >>> >> Brayan Perera > >>> >> > >>> >> > >>> > >>> > >>> > >>> -- > >>> Best Regards, > >>> Brayan Perera > >>> > > > > > > > > -- > > Best Regards, > > Brayan Perera > > > > -- > Best Regards, > Brayan Perera > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
