rados -p igbhome_data listomapkeys 100.00000000 stray0_head stray1_head stray2_head stray3_head stray4_head stray5_head stray6_head stray8_head stray9_head On 10/26/2017 05:08 AM, Daniel Davidson wrote:
I increased the logging of the mds to try and get some more information. I think the relevant lines are:2017-10-26 05:03:17.661683 7f1c598a6700 0 mds.0.cache.dir(607) _fetched missing object for [dir 607 ~mds0/stray7/ [2,head] auth v=108918871 cv=0/0 ap=1+0+0 state=1610645632 f(v1 m2017-10-25 14:56:13.140995 299= 299+0) n(v1 rc2017-10-25 14:56:13.140995 b191590453903 299=299+0) hs=0+11,ss=0+0 dirty=11 | child=1 sticky=1 dirty=1 waiter=1 authpin=1 0x7f1c71e9f300] 2017-10-26 05:03:17.661708 7f1c598a6700 -1 log_channel(cluster) log [ERR] : dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-26 05:03:17.661711 7f1c598a6700 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 607 is fatal because it is a system directory for this rankI would be grateful for any help in repair, Dan On 10/25/2017 04:17 PM, Daniel Davidson wrote:A bit more news: I made the ceph-0 mds shut down, started the mds on ceph-1 and then told it the mds was repaired. Everything ran great for about 5 hours and now it has crashed again. Same error:2017-10-25 15:13:07.344093 mon.0 [INF] fsmap e121828: 1/1/1 up {0=ceph-1=up:active} 2017-10-25 15:13:07.383445 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-25 15:13:07.480785 mon.0 [INF] osdmap e35296: 32 osds: 32 up, 32 in 2017-10-25 15:13:07.530337 mon.0 [INF] pgmap v28449919: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 5894 kB/s rd, 26 op/s 2017-10-25 15:13:08.473363 mon.0 [INF] mds.0 172.16.31.2:6802/3109594408 down:damaged 2017-10-25 15:13:08.473487 mon.0 [INF] fsmap e121829: 0/1/1 up, 1 damagedIf I: # rados -p igbhome_data rmomapkey 100.00000000 stray7_head # ceph mds repaired 0 then I get:2017-10-25 16:11:52.219916 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-25 16:11:52.307975 mon.0 [INF] osdmap e35322: 32 osds: 32 up, 32 in 2017-10-25 16:11:52.357904 mon.0 [INF] pgmap v28450567: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 11773 kB/s rd, 5262 kB/s wr, 26 op/s 2017-10-25 16:11:53.325331 mon.0 [INF] mds.0 172.16.31.2:6802/2716803172 down:damaged 2017-10-25 16:11:53.325424 mon.0 [INF] fsmap e121882: 0/1/1 up, 1 damaged 2017-10-25 16:11:53.475087 mon.0 [INF] pgmap v28450568: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 39677 kB/s rd, 47236 B/s wr, 54 op/s 2017-10-25 16:11:54.590232 mon.0 [INF] pgmap v28450569: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 28105 kB/s rd, 3786 kB/s wr, 43 op/s 2017-10-25 16:11:55.719476 mon.0 [INF] pgmap v28450570: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 26284 kB/s rd, 3678 kB/s wr, 357 op/s 2017-10-25 16:11:56.830623 mon.0 [INF] pgmap v28450571: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 37249 kB/s rd, 5476 B/s wr, 358 op/s 2017-10-25 16:11:57.965330 mon.0 [INF] pgmap v28450572: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 60769 kB/s rd, 53485 B/s wr, 41 op/s 2017-10-25 16:11:58.033787 mon.0 [INF] mds.? 172.16.31.2:6802/2725942008 up:boot 2017-10-25 16:11:58.033876 mon.0 [INF] fsmap e121883: 0/1/1 up, 1 up:standby, 1 damagedDan On 10/25/2017 11:30 AM, Daniel Davidson wrote:The system is down again saying it is missing the same stray7 again.2017-10-25 11:24:29.736774 mds.0 [WRN] failed to reconnect caps for missing inodes:2017-10-25 11:24:29.736779 mds.0 [WRN] ino 100147160e62017-10-25 11:24:29.753665 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7)Dan On 10/25/2017 08:54 AM, Daniel Davidson wrote:Thanks for the information. I did: # ceph daemon mds.ceph-0 scrub_path / repair recursive Saw in the logs it finished # ceph daemon mds.ceph-0 flush journal Saw in the logs it finished #ceph mds fail 0 #ceph mds repaired 0And it went back to missing stray7 again. I added that back like we did earlier and the system is back on line again, but the metadata errors still exist.Dan On 10/25/2017 07:50 AM, John Spray wrote:Commands that start "ceph daemon" take mds.<name> rather than a rank (notes on terminology here: http://docs.ceph.com/docs/master/cephfs/standby/). The name is how you would refer to the daemon from systemd, it's often set to the hostname where the daemon is running by default. John On Wed, Oct 25, 2017 at 2:30 PM, Daniel Davidson <danield@xxxxxxxxxxxxxxxx> wrote:I do have a problem with running the commands you mentioned to repair themds: # ceph daemon mds.0 scrub_pathadmin_socket: exception getting command descriptions: [Errno 2] No such fileor directoryadmin_socket: exception getting command descriptions: [Errno 2] No such fileor directory Any idea why that is not working? Dan On 10/25/2017 06:45 AM, Daniel Davidson wrote:John, thank you so much. After doing the initial rados command youmentioned it is back up and running. It did complain about a bunch of files which frankly are not important having duplicate inodes, but I will run those repair and scrub commands you mentioned and get it back clean again.Dan On 10/25/2017 03:55 AM, John Spray wrote:On Tue, Oct 24, 2017 at 7:14 PM, Daniel Davidson <danield@xxxxxxxxxxxxxxxx> wrote:OK, so "fixed" might be a bit of an overstatement here: while the PG is considered healthy, from CephFS's point of view what happened wasOur ceph system is having a problem.A few days a go we had a pg that was marked as inconsistent, and today Ifixed it with a: #ceph pg repair 1.37c then a file was stuck as missing so I did a: #ceph pg 1.37c mark_unfound_lost delete pg has 1 objects unfound and apparently lost markingthat some of its metadata just got blown away.There are some (most) objects that CephFS can do without (it will just EIO when you try to read that file/dir), but there are some that are essential and will cause a whole MDS rank to be damaged (unstartable)-- that's what's happened in your case.That fixed the unfound file problem and all the pgs went active+clean.Afew minutes later though, the FS seemed to pause and the MDS startedgiving errors. # ceph -w cluster 7bffce86-9d7b-4bdf-a9c9-67670e68ca77 health HEALTH_ERR mds rank 0 is damaged mds cluster is degraded noscrub,nodeep-scrub flag(s) set monmap e3: 4 mons at{ceph-0=172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0}election epoch 652, quorum 0,1,2,3 ceph-0,ceph-1,ceph-2,ceph-3 fsmap e121409: 0/1/1 up, 4 up:standby, 1 damaged osdmap e35220: 32 osds: 32 up, 32 inflags noscrub,nodeep-scrub,sortbitwise,require_jewel_osds pgmap v28398840: 1536 pgs, 2 pools, 795 TB data, 329 Mobjects1595 TB used, 1024 TB / 2619 TB avail 1536 active+clean Looking into the logs when I try a: #ceph mds repaired 0 2017-10-24 12:01:27.354271 mds.0 172.16.31.3:6801/1949050374 75 : cluster [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7) Any ideas as for what to do next, I am stumped.So if this is really the only missing object, then it's your lucky day, you lost a stray directory which usually contain just deletedfiles (can contain something more important if you've had hardlinkswhere the original file was later deleted).The MDS goes damaged if it has a reference to a stay directory, but the directory object isn't found. OTOH if there is no reference to the stray directory, it will happily recreate it for you. So, you cando this: rados -p <your metadata pool> rmomapkey 100.00000000 stray7_head...to prompt the MDS to recreate the stray directory (the argumentsthere are the magic internal names for ~mds0/stray7).Then, if that was the only damage, your MDS will come up after you run"ceph mds repaired 0". There will still be some inconsistency resulting from removing the stray dir, and possibly also from the disaster recovery tools that you've run since, so you'll want to do a "ceph daemon mds.<id>scrub_path / repair recursive". This will probably output a bunch of messages to the cluster log about things that it is repairing. Then do "ceph daemon mds.<id> flush journal" to flush out the repairs ithas made, and restart the MDS daemon one more time ("ceph mds fail 0"). JohnDan_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
