Hi, After the repair process, i have : 1926 active+clean 2 active+clean+inconsistent This two PGs seem to be on the same osd ( #34 ): # ceph pg dump | grep inconsistent dumped all in format plain 0.2e 4 0 0 0 8388660 4 4 active+clean+inconsistent 2014-07-16 11:39:43.819631 9463'4 438411:133968 [34,4] 34 [34,4] 34 9463'4 2014-07-16 04:52:54.417333 9463'4 2014-07-11 09:29:22.041717 0.1ed 5 0 0 0 8388623 10 10 active+clean+inconsistent 2014-07-16 11:39:45.820142 9712'10 438411:144792 [34,2] 34 [34,2] 34 9712'10 2014-07-16 09:12:44.742488 9712'10 2014-07-10 21:57:11.345241 It's can explain why my MDS won't to start ? If i remove ( or shutdown ) this OSD, it's can solved my problem ? Regards. Le 10/07/2014 11:51, Pierre BLONDEAU a ?crit : > Hi, > > Great. > > All my OSD restart : > osdmap e438044: 36 osds: 36 up, 36 in > > All PG page are active and some in recovery : > 1604040/49575206 objects degraded (3.236%) > 1780 active+clean > 17 active+degraded+remapped+backfilling > 61 active+degraded+remapped+wait_backfill > 11 active+clean+scrubbing+deep > 34 active+remapped+backfilling > 21 active+remapped+wait_backfill > 4 active+clean+replay > > But all mds crash. Logs are here : > https://blondeau.users.greyc.fr/cephlog/legacy/ > > In any case, thank you very much for your help. > > Pierre > > Le 09/07/2014 19:34, Joao Eduardo Luis a ?crit : >> On 07/09/2014 02:22 PM, Pierre BLONDEAU wrote: >>> Hi, >>> >>> There is any chance to restore my data ? >> >> Okay, I talked to Sam and here's what you could try before anything else: >> >> - Make sure you have everything running on the same version. >> - unset the the chooseleaf_vary_r flag -- this can be accomplished by >> setting tunables to legacy. >> - have the osds join in the cluster >> - you should then either upgrade to firefly (if you haven't done so by >> now) or wait for the point-release before you move on to setting >> tunables to optimal again. >> >> Let us know how it goes. >> >> -Joao >> >> >>> >>> Regards >>> Pierre >>> >>> Le 07/07/2014 15:42, Pierre BLONDEAU a ?crit : >>>> No chance to have those logs and even less in debug mode. I do this >>>> change 3 weeks ago. >>>> >>>> I put all my log here if it's can help : >>>> https://blondeau.users.greyc.fr/cephlog/all/ >>>> >>>> I have a chance to recover my +/- 20TB of data ? >>>> >>>> Regards >>>> >>>> Le 03/07/2014 21:48, Joao Luis a ?crit : >>>>> Do those logs have a higher debugging level than the default? If not >>>>> nevermind as they will not have enough information. If they do >>>>> however, >>>>> we'd be interested in the portion around the moment you set the >>>>> tunables. Say, before the upgrade and a bit after you set the tunable. >>>>> If you want to be finer grained, then ideally it would be the moment >>>>> where those maps were created, but you'd have to grep the logs for >>>>> that. >>>>> >>>>> Or drop the logs somewhere and I'll take a look. >>>>> >>>>> -Joao >>>>> >>>>> On Jul 3, 2014 5:48 PM, "Pierre BLONDEAU" <pierre.blondeau at unicaen.fr >>>>> <mailto:pierre.blondeau at unicaen.fr>> wrote: >>>>> >>>>> Le 03/07/2014 13:49, Joao Eduardo Luis a ?crit : >>>>> >>>>> On 07/03/2014 12:15 AM, Pierre BLONDEAU wrote: >>>>> >>>>> Le 03/07/2014 00:55, Samuel Just a ?crit : >>>>> >>>>> Ah, >>>>> >>>>> ~/logs ? for i in 20 23; do ../ceph/src/osdmaptool >>>>> --export-crush >>>>> /tmp/crush$i osd-$i*; ../ceph/src/crushtool -d >>>>> /tmp/crush$i > >>>>> /tmp/crush$i.d; done; diff /tmp/crush20.d >>>>> /tmp/crush23.d >>>>> ../ceph/src/osdmaptool: osdmap file >>>>> 'osd-20_osdmap.13258__0___4E62BB79__none' >>>>> ../ceph/src/osdmaptool: exported crush map to >>>>> /tmp/crush20 >>>>> ../ceph/src/osdmaptool: osdmap file >>>>> 'osd-23_osdmap.13258__0___4E62BB79__none' >>>>> ../ceph/src/osdmaptool: exported crush map to >>>>> /tmp/crush23 >>>>> 6d5 >>>>> < tunable chooseleaf_vary_r 1 >>>>> >>>>> Looks like the chooseleaf_vary_r tunable somehow >>>>> ended >>>>> up divergent? >>>>> >>>>> >>>>> The only thing that comes to mind that could cause this is >>>>> if we >>>>> changed >>>>> the leader's in-memory map, proposed it, it failed, and only >>>>> the >>>>> leader >>>>> got to write the map to disk somehow. This happened once on a >>>>> totally >>>>> different issue (although I can't pinpoint right now which). >>>>> >>>>> In such a scenario, the leader would serve the incorrect >>>>> osdmap to >>>>> whoever asked osdmaps from it, the remaining quorum would >>>>> serve the >>>>> correct osdmaps to all the others. This could cause this >>>>> divergence. Or >>>>> it could be something else. >>>>> >>>>> Are there logs for the monitors for the timeframe this may >>>>> have >>>>> happened >>>>> in? >>>>> >>>>> >>>>> Which exactly timeframe you want ? I have 7 days of logs, I should >>>>> have informations about the upgrade from firefly to 0.82. >>>>> Which mon's log do you want ? Three ? >>>>> >>>>> Regards >>>>> >>>>> -Joao >>>>> >>>>> >>>>> Pierre: do you recall how and when that got set? >>>>> >>>>> >>>>> I am not sure to understand, but if I good remember after >>>>> the update in >>>>> firefly, I was in state : HEALTH_WARN crush map has legacy >>>>> tunables and >>>>> I see "feature set mismatch" in log. >>>>> >>>>> So if I good remeber, i do : ceph osd crush tunables >>>>> optimal >>>>> for the >>>>> problem of "crush map" and I update my client and server >>>>> kernel to >>>>> 3.16rc. >>>>> >>>>> It's could be that ? >>>>> >>>>> Pierre >>>>> >>>>> -Sam >>>>> >>>>> On Wed, Jul 2, 2014 at 3:43 PM, Samuel Just >>>>> <sam.just at inktank.com <mailto:sam.just at inktank.com>> >>>>> wrote: >>>>> >>>>> Yeah, divergent osdmaps: >>>>> 555ed048e73024687fc8b106a570db__4f >>>>> osd-20_osdmap.13258__0___4E62BB79__none >>>>> 6037911f31dc3c18b05499d24dcdbe__5c >>>>> osd-23_osdmap.13258__0___4E62BB79__none >>>>> >>>>> Joao: thoughts? >>>>> -Sam >>>>> >>>>> On Wed, Jul 2, 2014 at 3:39 PM, Pierre BLONDEAU >>>>> <pierre.blondeau at unicaen.fr >>>>> <mailto:pierre.blondeau at unicaen.fr>> wrote: >>>>> >>>>> The files >>>>> >>>>> When I upgrade : >>>>> ceph-deploy install --stable firefly >>>>> servers... >>>>> on each servers service ceph restart mon >>>>> on each servers service ceph restart osd >>>>> on each servers service ceph restart mds >>>>> >>>>> I upgraded from emperor to firefly. After >>>>> repair, remap, replace, >>>>> etc ... I >>>>> have some PG which pass in peering state. >>>>> >>>>> I thought why not try the version 0.82, it >>>>> could >>>>> solve my problem. ( >>>>> It's my mistake ). So, I upgrade from >>>>> firefly to >>>>> 0.83 with : >>>>> ceph-deploy install --testing servers... >>>>> .. >>>>> >>>>> Now, all programs are in version 0.82. >>>>> I have 3 mons, 36 OSD and 3 mds. >>>>> >>>>> Pierre >>>>> >>>>> PS : I find also >>>>> "inc\uosdmap.13258__0___469271DE__none" on >>>>> each meta >>>>> directory. >>>>> >>>>> Le 03/07/2014 00:10, Samuel Just a ?crit : >>>>> >>>>> Also, what version did you upgrade from, >>>>> and >>>>> how did you upgrade? >>>>> -Sam >>>>> >>>>> On Wed, Jul 2, 2014 at 3:09 PM, Samuel >>>>> Just >>>>> <sam.just at inktank.com >>>>> <mailto:sam.just at inktank.com>> >>>>> wrote: >>>>> >>>>> >>>>> Ok, in current/meta on osd 20 and osd >>>>> 23, please attach all files >>>>> matching >>>>> >>>>> ^osdmap.13258.* >>>>> >>>>> There should be one such file on each >>>>> osd. (should look something >>>>> like >>>>> osdmap.6__0_FD6E4C01__none, probably >>>>> hashed into a subdirectory, >>>>> you'll want to use find). >>>>> >>>>> What version of ceph is running on >>>>> your >>>>> mons? How many mons do >>>>> you have? >>>>> -Sam >>>>> >>>>> On Wed, Jul 2, 2014 at 2:21 PM, Pierre >>>>> BLONDEAU >>>>> <pierre.blondeau at unicaen.fr >>>>> <mailto:pierre.blondeau at unicaen.fr>> >>>>> wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> I do it, the log files are >>>>> available >>>>> here : >>>>> >>>>> https://blondeau.users.greyc.__fr/cephlog/debug20/ >>>>> >>>>> <https://blondeau.users.greyc.fr/cephlog/debug20/> >>>>> >>>>> The OSD's files are really big +/- >>>>> 80M . >>>>> >>>>> After starting the osd.20 some >>>>> other >>>>> osd crash. I pass from 31 >>>>> osd up to >>>>> 16. >>>>> I remark that after this the >>>>> number >>>>> of down+peering PG decrease >>>>> from 367 >>>>> to >>>>> 248. It's "normal" ? May be it's >>>>> temporary, the time that the >>>>> cluster >>>>> verifies all the PG ? >>>>> >>>>> Regards >>>>> Pierre >>>>> >>>>> Le 02/07/2014 19:16, Samuel Just a >>>>> ?crit : >>>>> >>>>> You should add >>>>> >>>>> debug osd = 20 >>>>> debug filestore = 20 >>>>> debug ms = 1 >>>>> >>>>> to the [osd] section of the >>>>> ceph.conf and restart the >>>>> osds. I'd >>>>> like >>>>> all three logs if possible. >>>>> >>>>> Thanks >>>>> -Sam >>>>> >>>>> On Wed, Jul 2, 2014 at 5:03 >>>>> AM, >>>>> Pierre BLONDEAU >>>>> <pierre.blondeau at unicaen.fr >>>>> >>>>> <mailto:pierre.blondeau at unicaen.fr>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> Yes, but how i do that ? >>>>> >>>>> With a command like that ? >>>>> >>>>> ceph tell osd.20 >>>>> injectargs >>>>> '--debug-osd 20 >>>>> --debug-filestore 20 >>>>> --debug-ms >>>>> 1' >>>>> >>>>> By modify the >>>>> /etc/ceph/ceph.conf ? This >>>>> file is really poor >>>>> because I >>>>> use >>>>> udev detection. >>>>> >>>>> When I have made these >>>>> changes, you want the >>>>> three >>>>> log files or >>>>> only >>>>> osd.20's ? >>>>> >>>>> Thank you so much for the >>>>> help >>>>> >>>>> Regards >>>>> Pierre >>>>> >>>>> Le 01/07/2014 23:51, >>>>> Samuel >>>>> Just a ?crit : >>>>> >>>>> Can you reproduce with >>>>> debug osd = 20 >>>>> debug filestore = 20 >>>>> debug ms = 1 >>>>> ? >>>>> -Sam >>>>> >>>>> On Tue, Jul 1, 2014 at >>>>> 1:21 AM, Pierre >>>>> BLONDEAU >>>>> >>>>> <pierre.blondeau at unicaen.fr >>>>> >>>>> <mailto:pierre.blondeau at unicaen.fr>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> I join : >>>>> - osd.20 is >>>>> one of osd that I >>>>> detect which makes >>>>> crash >>>>> other >>>>> OSD. >>>>> - osd.23 is >>>>> one of osd which >>>>> crash when i start >>>>> osd.20 >>>>> - mds, is >>>>> one >>>>> of my MDS >>>>> >>>>> I cut log file >>>>> because they >>>>> are to >>>>> big but. All is >>>>> here : >>>>> >>>>> https://blondeau.users.greyc.__fr/cephlog/ >>>>> >>>>> <https://blondeau.users.greyc.fr/cephlog/> >>>>> >>>>> Regards >>>>> >>>>> Le 30/06/2014 >>>>> 17:35, >>>>> Gregory Farnum a >>>>> ?crit : >>>>> >>>>> What's the >>>>> backtrace from >>>>> the crashing >>>>> OSDs? >>>>> >>>>> Keep in mind >>>>> that as a dev >>>>> release, it's >>>>> generally best >>>>> not to >>>>> upgrade >>>>> to unnamed >>>>> versions like >>>>> 0.82 (but it's >>>>> probably too >>>>> late >>>>> to go >>>>> back >>>>> now). >>>>> >>>>> >>>>> >>>>> >>>>> I will remember it >>>>> the next time ;) >>>>> >>>>> -Greg >>>>> Software >>>>> Engineer #42 @ >>>>> >>>>> http://inktank.com >>>>> | >>>>> http://ceph.com >>>>> >>>>> On Mon, Jun >>>>> 30, >>>>> 2014 at 8:06 >>>>> AM, >>>>> Pierre >>>>> BLONDEAU >>>>> >>>>> <pierre.blondeau at unicaen.fr >>>>> >>>>> <mailto:pierre.blondeau at unicaen.fr>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> After the >>>>> upgrade to >>>>> firefly, I >>>>> have >>>>> some PG >>>>> in peering >>>>> state. >>>>> I seen the >>>>> output of >>>>> 0.82 so I >>>>> try to >>>>> upgrade >>>>> for >>>>> solved my >>>>> problem. >>>>> >>>>> My three >>>>> MDS >>>>> crash and >>>>> some OSD >>>>> triggers a >>>>> chain >>>>> reaction >>>>> that >>>>> kills >>>>> other >>>>> OSD. >>>>> I think my >>>>> MDS will >>>>> not >>>>> start >>>>> because of >>>>> the >>>>> metadata are >>>>> on the >>>>> OSD. >>>>> >>>>> I have 36 >>>>> OSD on >>>>> three >>>>> servers >>>>> and >>>>> I >>>>> identified >>>>> 5 OSD >>>>> which >>>>> makes >>>>> crash >>>>> others. >>>>> If i >>>>> not start >>>>> their, the >>>>> cluster >>>>> passe in >>>>> >>>>> reconstructive >>>>> state >>>>> with >>>>> 31 OSD >>>>> but i >>>>> have >>>>> 378 in >>>>> >>>>> down+peering >>>>> state. >>>>> >>>>> How can >>>>> I do >>>>> ? Would >>>>> you >>>>> more >>>>> >>>>> information >>>>> ( os, >>>>> crash log, >>>>> etc ... >>>>> ) >>>>> ? >>>>> >>>>> Regards >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> ------------------------------__---------------- >>>>> Pierre BLONDEAU >>>>> Administrateur Syst?mes & r?seaux >>>>> Universit? de Caen >>>>> Laboratoire GREYC, D?partement >>>>> d'informatique >>>>> >>>>> tel : 02 31 56 75 42 >>>>> bureau : Campus 2, Science 3, 406 >>>>> >>>>> ------------------------------__---------------- >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> ------------------------------__---------------- >>>>> Pierre BLONDEAU >>>>> Administrateur Syst?mes & r?seaux >>>>> Universit? de Caen >>>>> Laboratoire GREYC, D?partement d'informatique >>>>> >>>>> tel : 02 31 56 75 42 >>>>> bureau : Campus 2, Science 3, 406 >>>>> >>>>> ------------------------------__---------------- >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> ------------------------------__---------------- >>>>> Pierre BLONDEAU >>>>> Administrateur Syst?mes & r?seaux >>>>> Universit? de Caen >>>>> Laboratoire GREYC, D?partement d'informatique >>>>> >>>>> tel : 02 31 56 75 42 >>>>> bureau : Campus 2, Science 3, 406 >>>>> ------------------------------__---------------- >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users at lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>> >> >> > > -- ---------------------------------------------- Pierre BLONDEAU Administrateur Syst?mes & r?seaux Universit? de Caen Laboratoire GREYC, D?partement d'informatique tel : 02 31 56 75 42 bureau : Campus 2, Science 3, 406 ---------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2947 bytes Desc: Signature cryptographique S/MIME URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140716/938ec433/attachment.bin>