Re: mds0: Metadata damage detected

[John Spray <jspray@xxxxxxxxxx>]

There should be entries in the cluster log about this damage, so find them, and possibly use the timestamps to refer to the MDS log for more detail.

Was the cluster originally created with infernalis or earlier?  We have seen similar issues with a much older cluster that had actually lost dirfrag objects from a probable bug on older osds a long time ago -- the "upgrade" in jewel is that it actively records and reports this kind of thing.

You can use `find` with the inode numbers from damage ls to identify which paths are effected.

This is potentially repairable depending on exactly what's wrong.

John

On 24 Sep 2016 10:28 p.m., "Simion Marius Rad" <simarad@xxxxxxxxx> wrote:

Hi All, 

I run a CephFS production setup on Jewel 10.2.2 (updated from Infernalis 9.2.1) and recently my cluster status changed and now it show that metadata is damaged. I output of the "damage ls" command looks like this:

[{"damage_type":"dir_frag","id":9547171,"ino":1099624711247,"frag":"*"},{"damage_type":"dir_frag","id":34079746,"ino":1099597125430,"frag":"*"},{"damage_type":"dir_frag","id":52946895,"ino":1099604583518,"frag":"*"},{"damage_type":"dir_frag","id":148496486,"ino":1099624711225,"frag":"*"},{"damage_type":"dir_frag","id":200490809,"ino":1099625521620,"frag":"*"},{"damage_type":"dir_frag","id":205557997,"ino":1099606991149,"frag":"*"},{"damage_type":"dir_frag","id":258125406,"ino":1099618690902,"frag":"*"},{"damage_type":"dir_frag","id":510860270,"ino":1099618587040,"frag":"*"},{"damage_type":"dir_frag","id":549759027,"ino":1099628142420,"frag":"*"},{"damage_type":"dir_frag","id":585330960,"ino":1099624189139,"frag":"*"},{"damage_type":"dir_frag","id":820358338,"ino":1099624712699,"frag":"*"},{"damage_type":"dir_frag","id":942744318,"ino":1099628140501,"frag":"*"},{"damage_type":"dir_frag","id":973031988,"ino":1099549383938,"frag":"*"},{"damage_type":"dir_frag","id":1031321231,"ino":1099625361195,"frag":"*"},{"damage_type":"dir_frag","id":1045361182,"ino":1099608544391,"frag":"*"},{"damage_type":"dir_frag","id":1072944406,"ino":1099624188089,"frag":"*"},{"damage_type":"dir_frag","id":1074378376,"ino":1099624711044,"frag":"*"},{"damage_type":"dir_frag","id":1264544266,"ino":1099625355604,"frag":"*"},{"damage_type":"dir_frag","id":1265110397,"ino":1099618372284,"frag":"*"},{"damage_type":"dir_frag","id":1267294756,"ino":1099608269825,"frag":"*"},{"damage_type":"dir_frag","id":1443589698,"ino":1099628140270,"frag":"*"},{"damage_type":"dir_frag","id":1469860849,"ino":1099623182378,"frag":"*"},{"damage_type":"dir_frag","id":1630484148,"ino":1099620745725,"frag":"*"},{"damage_type":"dir_frag","id":1710015528,"ino":1099618586389,"frag":"*"},{"damage_type":"dir_frag","id":1723591292,"ino":1099609007327,"frag":"*"},{"damage_type":"dir_frag","id":1724584256,"ino":1099624712896,"frag":"*"},{"damage_type":"dir_frag","id":1806338857,"ino":1099622921186,"frag":"*"},{"damage_type":"dir_frag","id":1871119128,"ino":1099618123212,"frag":"*"},{"damage_type":"dir_frag","id":1886372044,"ino":1099609002836,"frag":"*"},{"damage_type":"dir_frag","id":1940442786,"ino":1099552476715,"frag":"*"},{"damage_type":"dir_frag","id":1984022309,"ino":1099622715593,"frag":"*"},{"damage_type":"dir_frag","id":2015183937,"ino":1099622864619,"frag":"*"},{"damage_type":"dir_frag","id":2019732391,"ino":1099618117660,"frag":"*"},{"damage_type":"dir_frag","id":2078183820,"ino":1099628250792,"frag":"*"},{"damage_type":"dir_frag","id":2086391724,"ino":1099607263818,"frag":"*"},{"damage_type":"dir_frag","id":2130427380,"ino":1099625363955,"frag":"*"},{"damage_type":"dir_frag","id":2256866899,"ino":1099618114228,"frag":"*"},{"damage_type":"dir_frag","id":2270858311,"ino":1099624713061,"frag":"*"},{"damage_type":"dir_frag","id":2317877380,"ino":1099608894918,"frag":"*"},{"damage_type":"dir_frag","id":2328690339,"ino":1099618586543,"frag":"*"},{"damage_type":"dir_frag","id":2352777654,"ino":1099624030883,"frag":"*"},{"damage_type":"dir_frag","id":2575139768,"ino":1099608266704,"frag":"*"},{"damage_type":"dir_frag","id":2630991415,"ino":1099628252432,"frag":"*"},{"damage_type":"dir_frag","id":2642152150,"ino":1099607934173,"frag":"*"},{"damage_type":"dir_frag","id":2812946108,"ino":1099607186041,"frag":"*"},{"damage_type":"dir_frag","id":2831890366,"ino":1099549384188,"frag":"*"},{"damage_type":"dir_frag","id":2885680303,"ino":1099622808358,"frag":"*"},{"damage_type":"dir_frag","id":3025465470,"ino":1099618587002,"frag":"*"},{"damage_type":"dir_frag","id":3056319918,"ino":1099624029233,"frag":"*"},{"damage_type":"dir_frag","id":3067128782,"ino":1099618586985,"frag":"*"},{"damage_type":"dir_frag","id":3103804005,"ino":1099622919068,"frag":"*"},{"damage_type":"dir_frag","id":3120240218,"ino":1099608666625,"frag":"*"},{"damage_type":"dir_frag","id":3253821953,"ino":1099607263367,"frag":"*"},{"damage_type":"dir_frag","id":3271496269,"ino":1099606721904,"frag":"*"},{"damage_type":"dir_frag","id":3394616906,"ino":1099621759230,"frag":"*"},{"damage_type":"dir_frag","id":3448326398,"ino":1099549383876,"frag":"*"},{"damage_type":"dir_frag","id":3527854419,"ino":1099628250053,"frag":"*"},{"damage_type":"dir_frag","id":3534906866,"ino":1099625726024,"frag":"*"},{"damage_type":"dir_frag","id":3599935903,"ino":1099607961924,"frag":"*"},{"damage_type":"dir_frag","id":3704836323,"ino":1099622920994,"frag":"*"},{"damage_type":"dir_frag","id":3744599290,"ino":1099618372557,"frag":"*"},{"damage_type":"dir_frag","id":3795019652,"ino":1099625520946,"frag":"*"},{"damage_type":"dir_frag","id":3804382536,"ino":1099609001266,"frag":"*"},{"damage_type":"dir_frag","id":3808069251,"ino":1099608786414,"frag":"*"},{"damage_type":"dir_frag","id":3853440337,"ino":1099627782649,"frag":"*"},{"damage_type":"dir_frag","id":3979835690,"ino":1099624029147,"frag":"*"},{"damage_type":"dir_frag","id":4008790181,"ino":1099609002882,"frag":"*"},{"damage_type":"dir_frag","id":4016298145,"ino":1099628250702,"frag":"*"},{"damage_type":"dir_frag","id":4030657237,"ino":1099623682582,"frag":"*"},{"damage_type":"dir_frag","id":4053237797,"ino":1099608787037,"frag":"*"}]

I also restarted the MDS and rerun an "ls -R" command and the status reappeared.

Is there any chance that I can fix this ?

Thank you,

Simion Rad

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com