Hello John and Greg, I used the new patch and now the undump succeeded and the mds is working fine and I can mount cephfs again! I still have one placement group which keeps deep scrubbing even after restarting the ceph cluster: dumped all in format plain 3.30 0 0 0 0 0 0 0 active+clean+scrubbing+deep 2014-11-10 17:21:15.866965 0'0 2414:418 [1,9] 1 [1,9] 1 631'3463 2014-08-21 15:14:45.430926 602'3131 2014-08-18 15:14:37.494913 I there a way to solve this? Kind regards, Jasper ________________________________________ Van: Gregory Farnum [greg@xxxxxxxxxxx] Verzonden: vrijdag 7 november 2014 22:42 Aan: Jasper Siero CC: ceph-users; John Spray Onderwerp: Re: mds isn't working anymore after osd's running full On Thu, Nov 6, 2014 at 11:49 AM, John Spray <john.spray@xxxxxxxxxx> wrote: > This is still an issue on master, so a fix will be coming soon. > Follow the ticket for updates: > http://tracker.ceph.com/issues/10025 > > Thanks for finding the bug! John is off for a vacation, but he pushed a branch wip-10025-firefly that if you install that (similar address to the other one) should work for you. You'll need to reset and undump again (I presume you still have the journal-as-a-file). I'll be merging them in to the stable branches pretty shortly as well. -Greg > > John > > On Thu, Nov 6, 2014 at 6:21 PM, John Spray <john.spray@xxxxxxxxxx> wrote: >> Jasper, >> >> Thanks for this -- I've reproduced this issue in a development >> environment. We'll see if this is also an issue on giant, and >> backport a fix if appropriate. I'll update this thread soon. >> >> Cheers, >> John >> >> On Mon, Nov 3, 2014 at 8:49 AM, Jasper Siero >> <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: >>> Hello Greg, >>> >>> I saw that the site of the previous link of the logs uses a very short expiring time so I uploaded it to another one: >>> >>> http://www.mediafire.com/download/gikiy7cqs42cllt/ceph-mds.th1-mon001.log.tar.gz >>> >>> Thanks, >>> >>> Jasper >>> >>> ________________________________________ >>> Van: gregory.farnum@xxxxxxxxxxx [gregory.farnum@xxxxxxxxxxx] namens Gregory Farnum [gfarnum@xxxxxxxxxx] >>> Verzonden: donderdag 30 oktober 2014 1:03 >>> Aan: Jasper Siero >>> CC: John Spray; ceph-users >>> Onderwerp: Re: mds isn't working anymore after osd's running full >>> >>> On Wed, Oct 29, 2014 at 7:51 AM, Jasper Siero >>> <jasper.siero@xxxxxxxxxxxxxxxxx> wrote: >>>> Hello Greg, >>>> >>>> I added the debug options which you mentioned and started the process again: >>>> >>>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 --pid-file /var/run/ceph/mds.th1-mon001.pid -c /etc/ceph/ceph.conf --cluster ceph --reset-journal 0 >>>> old journal was 9483323613~134233517 >>>> new journal start will be 9621733376 (4176246 bytes past old end) >>>> writing journal head >>>> writing EResetJournal entry >>>> done >>>> [root@th1-mon001 ~]# /usr/bin/ceph-mds -i th1-mon001 -c /etc/ceph/ceph.conf --cluster ceph --undump-journal 0 journaldumptgho-mon001 >>>> undump journaldumptgho-mon001 >>>> start 9483323613 len 134213311 >>>> writing header 200.00000000 >>>> writing 9483323613~1048576 >>>> writing 9484372189~1048576 >>>> writing 9485420765~1048576 >>>> writing 9486469341~1048576 >>>> writing 9487517917~1048576 >>>> writing 9488566493~1048576 >>>> writing 9489615069~1048576 >>>> writing 9490663645~1048576 >>>> writing 9491712221~1048576 >>>> writing 9492760797~1048576 >>>> writing 9493809373~1048576 >>>> writing 9494857949~1048576 >>>> writing 9495906525~1048576 >>>> writing 9496955101~1048576 >>>> writing 9498003677~1048576 >>>> writing 9499052253~1048576 >>>> writing 9500100829~1048576 >>>> writing 9501149405~1048576 >>>> writing 9502197981~1048576 >>>> writing 9503246557~1048576 >>>> writing 9504295133~1048576 >>>> writing 9505343709~1048576 >>>> writing 9506392285~1048576 >>>> writing 9507440861~1048576 >>>> writing 9508489437~1048576 >>>> writing 9509538013~1048576 >>>> writing 9510586589~1048576 >>>> writing 9511635165~1048576 >>>> writing 9512683741~1048576 >>>> writing 9513732317~1048576 >>>> writing 9514780893~1048576 >>>> writing 9515829469~1048576 >>>> writing 9516878045~1048576 >>>> writing 9517926621~1048576 >>>> writing 9518975197~1048576 >>>> writing 9520023773~1048576 >>>> writing 9521072349~1048576 >>>> writing 9522120925~1048576 >>>> writing 9523169501~1048576 >>>> writing 9524218077~1048576 >>>> writing 9525266653~1048576 >>>> writing 9526315229~1048576 >>>> writing 9527363805~1048576 >>>> writing 9528412381~1048576 >>>> writing 9529460957~1048576 >>>> writing 9530509533~1048576 >>>> writing 9531558109~1048576 >>>> writing 9532606685~1048576 >>>> writing 9533655261~1048576 >>>> writing 9534703837~1048576 >>>> writing 9535752413~1048576 >>>> writing 9536800989~1048576 >>>> writing 9537849565~1048576 >>>> writing 9538898141~1048576 >>>> writing 9539946717~1048576 >>>> writing 9540995293~1048576 >>>> writing 9542043869~1048576 >>>> writing 9543092445~1048576 >>>> writing 9544141021~1048576 >>>> writing 9545189597~1048576 >>>> writing 9546238173~1048576 >>>> writing 9547286749~1048576 >>>> writing 9548335325~1048576 >>>> writing 9549383901~1048576 >>>> writing 9550432477~1048576 >>>> writing 9551481053~1048576 >>>> writing 9552529629~1048576 >>>> writing 9553578205~1048576 >>>> writing 9554626781~1048576 >>>> writing 9555675357~1048576 >>>> writing 9556723933~1048576 >>>> writing 9557772509~1048576 >>>> writing 9558821085~1048576 >>>> writing 9559869661~1048576 >>>> writing 9560918237~1048576 >>>> writing 9561966813~1048576 >>>> writing 9563015389~1048576 >>>> writing 9564063965~1048576 >>>> writing 9565112541~1048576 >>>> writing 9566161117~1048576 >>>> writing 9567209693~1048576 >>>> writing 9568258269~1048576 >>>> writing 9569306845~1048576 >>>> writing 9570355421~1048576 >>>> writing 9571403997~1048576 >>>> writing 9572452573~1048576 >>>> writing 9573501149~1048576 >>>> writing 9574549725~1048576 >>>> writing 9575598301~1048576 >>>> writing 9576646877~1048576 >>>> writing 9577695453~1048576 >>>> writing 9578744029~1048576 >>>> writing 9579792605~1048576 >>>> writing 9580841181~1048576 >>>> writing 9581889757~1048576 >>>> writing 9582938333~1048576 >>>> writing 9583986909~1048576 >>>> writing 9585035485~1048576 >>>> writing 9586084061~1048576 >>>> writing 9587132637~1048576 >>>> writing 9588181213~1048576 >>>> writing 9589229789~1048576 >>>> writing 9590278365~1048576 >>>> writing 9591326941~1048576 >>>> writing 9592375517~1048576 >>>> writing 9593424093~1048576 >>>> writing 9594472669~1048576 >>>> writing 9595521245~1048576 >>>> writing 9596569821~1048576 >>>> writing 9597618397~1048576 >>>> writing 9598666973~1048576 >>>> writing 9599715549~1048576 >>>> writing 9600764125~1048576 >>>> writing 9601812701~1048576 >>>> writing 9602861277~1048576 >>>> writing 9603909853~1048576 >>>> writing 9604958429~1048576 >>>> writing 9606007005~1048576 >>>> writing 9607055581~1048576 >>>> writing 9608104157~1048576 >>>> writing 9609152733~1048576 >>>> writing 9610201309~1048576 >>>> writing 9611249885~1048576 >>>> writing 9612298461~1048576 >>>> writing 9613347037~1048576 >>>> writing 9614395613~1048576 >>>> writing 9615444189~1048576 >>>> writing 9616492765~1044159 >>>> done. >>>> [root@th1-mon001 ~]# service ceph start mds >>>> === mds.th1-mon001 === >>>> Starting Ceph mds.th1-mon001 on th1-mon001... >>>> starting mds.th1-mon001 at :/0 >>>> >>>> >>>> The new logs: >>>> http://pastebin.com/wqqjuEpy >>> >>> These don't have the increased debugging levels set. :( I'm not sure >>> where you could have put them that they didn't get picked up, but make >>> sure it's in the ceph.conf that this mds daemon is referring to. (You >>> can see the debug levels in use in the "--- logging levels ---" >>> section; they appear to all be default.) >>> -Greg >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |