Hi, We had applied the patch and recompile ceph as well as updated the ceph.conf as per suggested, when we re-run ceph-mds we noticed the following: 2014-04-29 10:45:22.260798 7f90b971d700 0 log [WRN] : replayed op client.324186:51366457,12681393 no session for client.324186 2014-04-29 10:45:22.262419 7f90b971d700 0 log [WRN] : replayed op client.324186:51366475,12681393 no session for client.324186 2014-04-29 10:45:22.267699 7f90b971d700 0 log [WRN] : replayed op client.324186:51366665,12681393 no session for client.324186 2014-04-29 10:45:22.271664 7f90b971d700 0 log [WRN] : replayed op client.324186:51366724,12681393 no session for client.324186 2014-04-29 10:45:22.281050 7f90b971d700 0 log [WRN] : replayed op client.324186:51366945,12681393 no session for client.324186 2014-04-29 10:45:22.283196 7f90b971d700 0 log [WRN] : replayed op client.324186:51366996,12681393 no session for client.324186 2014-04-29 10:45:22.287801 7f90b971d700 0 log [WRN] : replayed op client.324186:51367043,12681393 no session for client.324186 2014-04-29 10:45:22.289967 7f90b971d700 0 log [WRN] : replayed op client.324186:51367082,12681393 no session for client.324186 2014-04-29 10:45:22.291026 7f90b971d700 0 log [WRN] : replayed op client.324186:51367110,12681393 no session for client.324186 2014-04-29 10:45:22.294459 7f90b971d700 0 log [WRN] : replayed op client.324186:51367192,12681393 no session for client.324186 2014-04-29 10:45:22.297228 7f90b971d700 0 log [WRN] : replayed op client.324186:51367257,12681393 no session for client.324186 2014-04-29 10:45:22.297477 7f90b971d700 0 log [WRN] : replayed op client.324186:51367264,12681393 no session for client.324186 tcmalloc: large alloc 1136660480 bytes == 0xb2019000 @ 0x7f90c2564da7 0x5bb9cb 0x5ac8eb 0x5b32f7 0x79ecd8 0x58cbed 0x7f90c231de9a 0x7f90c0cca3fd tcmalloc: large alloc 2273316864 bytes == 0x15d73d000 @ 0x7f90c2564da7 0x5bb9cb 0x5ac8eb 0x5b32f7 0x79ecd8 0x58cbed 0x7f90c231de9a 0x7f90c0cca3fd ceph -s shows that MDS up:replay, Also the messages above seemed to be repeating again after a while but with a different session number. Is there a way for us to determine that we are on the right track? Thanks. Regards, Luke On Sun, Apr 27, 2014 at 12:04 PM, Yan, Zheng <ukernel@xxxxxxxxx> wrote: > On Sat, Apr 26, 2014 at 9:56 AM, Jingyuan Luke <jyluke@xxxxxxxxx> wrote: >> Hi Greg, >> >> Actually our cluster is pretty empty, but we suspect we had a temporary >> network disconnection to one of our OSD, not sure if this caused the >> problem. >> >> Anyway we don't mind try the method you mentioned, how can we do that? >> > > compile ceph-mds with the attached patch. add a line "mds > wipe_sessions = 1" to the ceph.conf, > > Yan, Zheng > >> Regards, >> Luke >> >> >> On Saturday, April 26, 2014, Gregory Farnum <greg@xxxxxxxxxxx> wrote: >>> >>> Hmm, it looks like your on-disk SessionMap is horrendously out of >>> date. Did your cluster get full at some point? >>> >>> In any case, we're working on tools to repair this now but they aren't >>> ready for use yet. Probably the only thing you could do is create an >>> empty sessionmap with a higher version than the ones the journal >>> refers to, but that might have other fallout effects... >>> -Greg >>> Software Engineer #42 @ http://inktank.com | http://ceph.com >>> >>> >>> On Fri, Apr 25, 2014 at 2:57 AM, Mohd Bazli Ab Karim >>> <bazli.abkarim@xxxxxxxx> wrote: >>> > More logs. I ran ceph-mds with debug-mds=20. >>> > >>> > -2> 2014-04-25 17:47:54.839672 7f0d6f3f0700 10 mds.0.journal >>> > EMetaBlob.replay inotable tablev 4316124 <= table 4317932 >>> > -1> 2014-04-25 17:47:54.839674 7f0d6f3f0700 10 mds.0.journal >>> > EMetaBlob.replay sessionmap v8632368 -(1|2) == table 7239603 prealloc >>> > [1000041df86~1] used 1000041db9e >>> > 0> 2014-04-25 17:47:54.840733 7f0d6f3f0700 -1 mds/journal.cc: In >>> > function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread >>> > 7f0d6f3f0700 time 2014-04-25 17:47:54.839688 mds/journal.cc: 1303: FAILED >>> > assert(session) >>> > >>> > Please look at the attachment for more details. >>> > >>> > Regards, >>> > Bazli >>> > >>> > From: Mohd Bazli Ab Karim >>> > Sent: Friday, April 25, 2014 12:26 PM >>> > To: 'ceph-devel@xxxxxxxxxxxxxxx'; ceph-users@xxxxxxxxxxxxxx >>> > Subject: Ceph mds laggy and failed assert in function replay >>> > mds/journal.cc >>> > >>> > Dear Ceph-devel, ceph-users, >>> > >>> > I am currently facing issue with my ceph mds server. Ceph-mds daemon >>> > does not want to bring up back. >>> > Tried running that manually with ceph-mds -i mon01 -d but it shows that >>> > it stucks at failed assert(session) line 1303 in mds/journal.cc and aborted. >>> > >>> > Can someone shed some light in this issue. >>> > ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) >>> > >>> > Let me know if I need to send log with debug enabled. >>> > >>> > Regards, >>> > Bazli >>> > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
