This is still an issue on master, so a fix will be coming soon. Follow the ticket for updates: http://tracker.ceph.com/issues/10025 Thanks for finding the bug! John On Thu, Nov 6, 2014 at 6:21 PM, John Spray <john.spray@xxxxxxxxxx> wrote: > Jasper, > > Thanks for this -- I've reproduced this issue in a development > environment. We'll see if this is also an issue on giant, and > backport a fix if appropriate. I'll update this thread soon. > > Cheers, > John > > On Mon, Nov 3, 2014 at 8:49 AM, Jasper Siero > <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: >> Hello Greg, >> >> I saw that the site of the previous link of the logs uses a very short expiring time so I uploaded it to another one: >> >> http://www.mediafire.com/download/gikiy7cqs42cllt/ceph-mds.th1-mon001.log.tar.gz >> >> Thanks, >> >> Jasper >> >> ________________________________________ >> Van: gregory.farnum@xxxxxxxxxxx [gregory.farnum@xxxxxxxxxxx] namens Gregory Farnum [gfarnum@xxxxxxxxxx] >> Verzonden: donderdag 30 oktober 2014 1:03 >> Aan: Jasper Siero >> CC: John Spray; ceph-users >> Onderwerp: Re: mds isn't working anymore after osd's running full >> >> On Wed, Oct 29, 2014 at 7:51 AM, Jasper Siero >> <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: >>> Hello Greg, >>> >>> I added the debug options which you mentioned and started the process again: >>> >>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph --reset-journal 0 >>> old journal was 9483323613~134233517 >>> new journal start will be 9621733376 (4176246 bytes past old end) >>> writing journal head >>> writing EResetJournal entry >>> done >>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 -c /etc/ceph/ceph.conf --cluster ceph --undump-journal 0 journaldumptgho-mon001 >>> undump journaldumptgho-mon001 >>> start 9483323613 len 134213311 >>> writing header 200.00000000 >>> writing 9483323613~1048576 >>> writing 9484372189~1048576 >>> writing 9485420765~1048576 >>> writing 9486469341~1048576 >>> writing 9487517917~1048576 >>> writing 9488566493~1048576 >>> writing 9489615069~1048576 >>> writing 9490663645~1048576 >>> writing 9491712221~1048576 >>> writing 9492760797~1048576 >>> writing 9493809373~1048576 >>> writing 9494857949~1048576 >>> writing 9495906525~1048576 >>> writing 9496955101~1048576 >>> writing 9498003677~1048576 >>> writing 9499052253~1048576 >>> writing 9500100829~1048576 >>> writing 9501149405~1048576 >>> writing 9502197981~1048576 >>> writing 9503246557~1048576 >>> writing 9504295133~1048576 >>> writing 9505343709~1048576 >>> writing 9506392285~1048576 >>> writing 9507440861~1048576 >>> writing 9508489437~1048576 >>> writing 9509538013~1048576 >>> writing 9510586589~1048576 >>> writing 9511635165~1048576 >>> writing 9512683741~1048576 >>> writing 9513732317~1048576 >>> writing 9514780893~1048576 >>> writing 9515829469~1048576 >>> writing 9516878045~1048576 >>> writing 9517926621~1048576 >>> writing 9518975197~1048576 >>> writing 9520023773~1048576 >>> writing 9521072349~1048576 >>> writing 9522120925~1048576 >>> writing 9523169501~1048576 >>> writing 9524218077~1048576 >>> writing 9525266653~1048576 >>> writing 9526315229~1048576 >>> writing 9527363805~1048576 >>> writing 9528412381~1048576 >>> writing 9529460957~1048576 >>> writing 9530509533~1048576 >>> writing 9531558109~1048576 >>> writing 9532606685~1048576 >>> writing 9533655261~1048576 >>> writing 9534703837~1048576 >>> writing 9535752413~1048576 >>> writing 9536800989~1048576 >>> writing 9537849565~1048576 >>> writing 9538898141~1048576 >>> writing 9539946717~1048576 >>> writing 9540995293~1048576 >>> writing 9542043869~1048576 >>> writing 9543092445~1048576 >>> writing 9544141021~1048576 >>> writing 9545189597~1048576 >>> writing 9546238173~1048576 >>> writing 9547286749~1048576 >>> writing 9548335325~1048576 >>> writing 9549383901~1048576 >>> writing 9550432477~1048576 >>> writing 9551481053~1048576 >>> writing 9552529629~1048576 >>> writing 9553578205~1048576 >>> writing 9554626781~1048576 >>> writing 9555675357~1048576 >>> writing 9556723933~1048576 >>> writing 9557772509~1048576 >>> writing 9558821085~1048576 >>> writing 9559869661~1048576 >>> writing 9560918237~1048576 >>> writing 9561966813~1048576 >>> writing 9563015389~1048576 >>> writing 9564063965~1048576 >>> writing 9565112541~1048576 >>> writing 9566161117~1048576 >>> writing 9567209693~1048576 >>> writing 9568258269~1048576 >>> writing 9569306845~1048576 >>> writing 9570355421~1048576 >>> writing 9571403997~1048576 >>> writing 9572452573~1048576 >>> writing 9573501149~1048576 >>> writing 9574549725~1048576 >>> writing 9575598301~1048576 >>> writing 9576646877~1048576 >>> writing 9577695453~1048576 >>> writing 9578744029~1048576 >>> writing 9579792605~1048576 >>> writing 9580841181~1048576 >>> writing 9581889757~1048576 >>> writing 9582938333~1048576 >>> writing 9583986909~1048576 >>> writing 9585035485~1048576 >>> writing 9586084061~1048576 >>> writing 9587132637~1048576 >>> writing 9588181213~1048576 >>> writing 9589229789~1048576 >>> writing 9590278365~1048576 >>> writing 9591326941~1048576 >>> writing 9592375517~1048576 >>> writing 9593424093~1048576 >>> writing 9594472669~1048576 >>> writing 9595521245~1048576 >>> writing 9596569821~1048576 >>> writing 9597618397~1048576 >>> writing 9598666973~1048576 >>> writing 9599715549~1048576 >>> writing 9600764125~1048576 >>> writing 9601812701~1048576 >>> writing 9602861277~1048576 >>> writing 9603909853~1048576 >>> writing 9604958429~1048576 >>> writing 9606007005~1048576 >>> writing 9607055581~1048576 >>> writing 9608104157~1048576 >>> writing 9609152733~1048576 >>> writing 9610201309~1048576 >>> writing 9611249885~1048576 >>> writing 9612298461~1048576 >>> writing 9613347037~1048576 >>> writing 9614395613~1048576 >>> writing 9615444189~1048576 >>> writing 9616492765~1044159 >>> done. >>> [root@th1-mon001 ~]# service ceph start mds >>> === mds.th1-mon001 === >>> Starting Ceph mds.th1-mon001 on th1-mon001... >>> starting mds.th1-mon001 at :/0 >>> >>> >>> The new logs: >>> http://pastebin.com/wqqjuEpy >> >> These don't have the increased debugging levels set. :( I'm not sure >> where you could have put them that they didn't get picked up, but make >> sure it's in the ceph.conf that this mds daemon is referring to. (You >> can see the debug levels in use in the "--- logging levels ---" >> section; they appear to all be default.) >> -Greg >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |