Hi, Assuming we got MDS working back on track, should we still leave the mds_wipe_sessions in the ceph.conf or remove it and restart MDS. Thanks. Regards, Luke On Tue, Apr 29, 2014 at 2:12 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: > On Tue, Apr 29, 2014 at 11:24 AM, Jingyuan Luke <jyluke@xxxxxxxxx> wrote: >> Hi, >> >> We had applied the patch and recompile ceph as well as updated the >> ceph.conf as per suggested, when we re-run ceph-mds we noticed the >> following: >> >> >> 2014-04-29 10:45:22.260798 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366457,12681393 no session for client.324186 >> 2014-04-29 10:45:22.262419 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366475,12681393 no session for client.324186 >> 2014-04-29 10:45:22.267699 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366665,12681393 no session for client.324186 >> 2014-04-29 10:45:22.271664 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366724,12681393 no session for client.324186 >> 2014-04-29 10:45:22.281050 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366945,12681393 no session for client.324186 >> 2014-04-29 10:45:22.283196 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51366996,12681393 no session for client.324186 >> 2014-04-29 10:45:22.287801 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367043,12681393 no session for client.324186 >> 2014-04-29 10:45:22.289967 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367082,12681393 no session for client.324186 >> 2014-04-29 10:45:22.291026 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367110,12681393 no session for client.324186 >> 2014-04-29 10:45:22.294459 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367192,12681393 no session for client.324186 >> 2014-04-29 10:45:22.297228 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367257,12681393 no session for client.324186 >> 2014-04-29 10:45:22.297477 7f90b971d700 0 log [WRN] : replayed op >> client.324186:51367264,12681393 no session for client.324186 >> >> tcmalloc: large alloc 1136660480 bytes == 0xb2019000 @ 0x7f90c2564da7 >> 0x5bb9cb 0x5ac8eb 0x5b32f7 0x79ecd8 0x58cbed 0x7f90c231de9a >> 0x7f90c0cca3fd >> tcmalloc: large alloc 2273316864 bytes == 0x15d73d000 @ >> 0x7f90c2564da7 0x5bb9cb 0x5ac8eb 0x5b32f7 0x79ecd8 0x58cbed >> 0x7f90c231de9a 0x7f90c0cca3fd >> >> ceph -s shows that MDS up:replay, >> >> Also the messages above seemed to be repeating again after a while but >> with a different session number. Is there a way for us to determine >> that we are on the right track? Thanks. >> > > It's on the right track as long as the MDS doesn't crash. > >> Regards, >> Luke >> >> On Sun, Apr 27, 2014 at 12:04 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: >>> On Sat, Apr 26, 2014 at 9:56 AM, Jingyuan Luke <jyluke@xxxxxxxxx> wrote: >>>> Hi Greg, >>>> >>>> Actually our cluster is pretty empty, but we suspect we had a temporary >>>> network disconnection to one of our OSD, not sure if this caused the >>>> problem. >>>> >>>> Anyway we don't mind try the method you mentioned, how can we do that? >>>> >>> >>> compile ceph-mds with the attached patch. add a line "mds >>> wipe_sessions = 1" to the ceph.conf, >>> >>> Yan, Zheng >>> >>>> Regards, >>>> Luke >>>> >>>> >>>> On Saturday, April 26, 2014, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>>>> >>>>> Hmm, it looks like your on-disk SessionMap is horrendously out of >>>>> date. Did your cluster get full at some point? >>>>> >>>>> In any case, we're working on tools to repair this now but they aren't >>>>> ready for use yet. Probably the only thing you could do is create an >>>>> empty sessionmap with a higher version than the ones the journal >>>>> refers to, but that might have other fallout effects... >>>>> -Greg >>>>> Software Engineer #42 @ http://inktank.com | http://ceph.com >>>>> >>>>> >>>>> On Fri, Apr 25, 2014 at 2:57 AM, Mohd Bazli Ab Karim >>>>> <bazli.abkarim@xxxxxxxx> wrote: >>>>> > More logs. I ran ceph-mds with debug-mds=20. >>>>> > >>>>> > -2> 2014-04-25 17:47:54.839672 7f0d6f3f0700 10 mds.0.journal >>>>> > EMetaBlob.replay inotable tablev 4316124 <= table 4317932 >>>>> > -1> 2014-04-25 17:47:54.839674 7f0d6f3f0700 10 mds.0.journal >>>>> > EMetaBlob.replay sessionmap v8632368 -(1|2) == table 7239603 prealloc >>>>> > [1000041df86~1] used 1000041db9e >>>>> > 0> 2014-04-25 17:47:54.840733 7f0d6f3f0700 -1 mds/journal.cc: In >>>>> > function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread >>>>> > 7f0d6f3f0700 time 2014-04-25 17:47:54.839688 mds/journal.cc: 1303: FAILED >>>>> > assert(session) >>>>> > >>>>> > Please look at the attachment for more details. >>>>> > >>>>> > Regards, >>>>> > Bazli >>>>> > >>>>> > From: Mohd Bazli Ab Karim >>>>> > Sent: Friday, April 25, 2014 12:26 PM >>>>> > To: 'ceph-devel@xxxxxxxxxxxxxxx'; ceph-users@xxxxxxxxxxxxxx >>>>> > Subject: Ceph mds laggy and failed assert in function replay >>>>> > mds/journal.cc >>>>> > >>>>> > Dear Ceph-devel, ceph-users, >>>>> > >>>>> > I am currently facing issue with my ceph mds server. Ceph-mds daemon >>>>> > does not want to bring up back. >>>>> > Tried running that manually with ceph-mds -i mon01 -d but it shows that >>>>> > it stucks at failed assert(session) line 1303 in mds/journal.cc and aborted. >>>>> > >>>>> > Can someone shed some light in this issue. >>>>> > ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) >>>>> > >>>>> > Let me know if I need to send log with debug enabled. >>>>> > >>>>> > Regards, >>>>> > Bazli >>>>> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
