Jasper, Thanks for this -- I've reproduced this issue in a development environment. We'll see if this is also an issue on giant, and backport a fix if appropriate. I'll update this thread soon. Cheers, John On Mon, Nov 3, 2014 at 8:49 AM, Jasper Siero <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: > Hello Greg, > > I saw that the site of the previous link of the logs uses a very short expiring time so I uploaded it to another one: > > http://www.mediafire.com/download/gikiy7cqs42cllt/ceph-mds.th1-mon001.log.tar.gz > > Thanks, > > Jasper > > ________________________________________ > Van: gregory.farnum@xxxxxxxxxxx [gregory.farnum@xxxxxxxxxxx] namens Gregory Farnum [gfarnum@xxxxxxxxxx] > Verzonden: donderdag 30 oktober 2014 1:03 > Aan: Jasper Siero > CC: John Spray; ceph-users > Onderwerp: Re: mds isn't working anymore after osd's running full > > On Wed, Oct 29, 2014 at 7:51 AM, Jasper Siero > <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: >> Hello Greg, >> >> I added the debug options which you mentioned and started the process again: >> >> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph --reset-journal 0 >> old journal was 9483323613~134233517 >> new journal start will be 9621733376 (4176246 bytes past old end) >> writing journal head >> writing EResetJournal entry >> done >> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 -c /etc/ceph/ceph.conf --cluster ceph --undump-journal 0 journaldumptgho-mon001 >> undump journaldumptgho-mon001 >> start 9483323613 len 134213311 >> writing header 200.00000000 >> writing 9483323613~1048576 >> writing 9484372189~1048576 >> writing 9485420765~1048576 >> writing 9486469341~1048576 >> writing 9487517917~1048576 >> writing 9488566493~1048576 >> writing 9489615069~1048576 >> writing 9490663645~1048576 >> writing 9491712221~1048576 >> writing 9492760797~1048576 >> writing 9493809373~1048576 >> writing 9494857949~1048576 >> writing 9495906525~1048576 >> writing 9496955101~1048576 >> writing 9498003677~1048576 >> writing 9499052253~1048576 >> writing 9500100829~1048576 >> writing 9501149405~1048576 >> writing 9502197981~1048576 >> writing 9503246557~1048576 >> writing 9504295133~1048576 >> writing 9505343709~1048576 >> writing 9506392285~1048576 >> writing 9507440861~1048576 >> writing 9508489437~1048576 >> writing 9509538013~1048576 >> writing 9510586589~1048576 >> writing 9511635165~1048576 >> writing 9512683741~1048576 >> writing 9513732317~1048576 >> writing 9514780893~1048576 >> writing 9515829469~1048576 >> writing 9516878045~1048576 >> writing 9517926621~1048576 >> writing 9518975197~1048576 >> writing 9520023773~1048576 >> writing 9521072349~1048576 >> writing 9522120925~1048576 >> writing 9523169501~1048576 >> writing 9524218077~1048576 >> writing 9525266653~1048576 >> writing 9526315229~1048576 >> writing 9527363805~1048576 >> writing 9528412381~1048576 >> writing 9529460957~1048576 >> writing 9530509533~1048576 >> writing 9531558109~1048576 >> writing 9532606685~1048576 >> writing 9533655261~1048576 >> writing 9534703837~1048576 >> writing 9535752413~1048576 >> writing 9536800989~1048576 >> writing 9537849565~1048576 >> writing 9538898141~1048576 >> writing 9539946717~1048576 >> writing 9540995293~1048576 >> writing 9542043869~1048576 >> writing 9543092445~1048576 >> writing 9544141021~1048576 >> writing 9545189597~1048576 >> writing 9546238173~1048576 >> writing 9547286749~1048576 >> writing 9548335325~1048576 >> writing 9549383901~1048576 >> writing 9550432477~1048576 >> writing 9551481053~1048576 >> writing 9552529629~1048576 >> writing 9553578205~1048576 >> writing 9554626781~1048576 >> writing 9555675357~1048576 >> writing 9556723933~1048576 >> writing 9557772509~1048576 >> writing 9558821085~1048576 >> writing 9559869661~1048576 >> writing 9560918237~1048576 >> writing 9561966813~1048576 >> writing 9563015389~1048576 >> writing 9564063965~1048576 >> writing 9565112541~1048576 >> writing 9566161117~1048576 >> writing 9567209693~1048576 >> writing 9568258269~1048576 >> writing 9569306845~1048576 >> writing 9570355421~1048576 >> writing 9571403997~1048576 >> writing 9572452573~1048576 >> writing 9573501149~1048576 >> writing 9574549725~1048576 >> writing 9575598301~1048576 >> writing 9576646877~1048576 >> writing 9577695453~1048576 >> writing 9578744029~1048576 >> writing 9579792605~1048576 >> writing 9580841181~1048576 >> writing 9581889757~1048576 >> writing 9582938333~1048576 >> writing 9583986909~1048576 >> writing 9585035485~1048576 >> writing 9586084061~1048576 >> writing 9587132637~1048576 >> writing 9588181213~1048576 >> writing 9589229789~1048576 >> writing 9590278365~1048576 >> writing 9591326941~1048576 >> writing 9592375517~1048576 >> writing 9593424093~1048576 >> writing 9594472669~1048576 >> writing 9595521245~1048576 >> writing 9596569821~1048576 >> writing 9597618397~1048576 >> writing 9598666973~1048576 >> writing 9599715549~1048576 >> writing 9600764125~1048576 >> writing 9601812701~1048576 >> writing 9602861277~1048576 >> writing 9603909853~1048576 >> writing 9604958429~1048576 >> writing 9606007005~1048576 >> writing 9607055581~1048576 >> writing 9608104157~1048576 >> writing 9609152733~1048576 >> writing 9610201309~1048576 >> writing 9611249885~1048576 >> writing 9612298461~1048576 >> writing 9613347037~1048576 >> writing 9614395613~1048576 >> writing 9615444189~1048576 >> writing 9616492765~1044159 >> done. >> [root@th1-mon001 ~]# service ceph start mds >> === mds.th1-mon001 === >> Starting Ceph mds.th1-mon001 on th1-mon001... >> starting mds.th1-mon001 at :/0 >> >> >> The new logs: >> http://pastebin.com/wqqjuEpy > > These don't have the increased debugging levels set. :( I'm not sure > where you could have put them that they didn't get picked up, but make > sure it's in the ceph.conf that this mds daemon is referring to. (You > can see the debug levels in use in the "--- logging levels ---" > section; they appear to all be default.) > -Greg > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |