Re: MDS damaged

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I increased the logging of the mds to try and get some more information.  I think the relevant lines are:

2017-10-26 05:03:17.661683 7f1c598a6700  0 mds.0.cache.dir(607) _fetched missing object for [dir 607 ~mds0/stray7/ [2,head] auth v=108918871 cv=0/0 ap=1+0+0 state=1610645632 f(v1 m2017-10-25 14:56:13.140995 299= 299+0) n(v1 rc2017-10-25 14:56:13.140995 b191590453903 299=299+0) hs=0+11,ss=0+0 dirty=11 | child=1 sticky=1 dirty=1 waiter=1 authpin=1 0x7f1c71e9f300] 2017-10-26 05:03:17.661708 7f1c598a6700 -1 log_channel(cluster) log [ERR] : dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-26 05:03:17.661711 7f1c598a6700 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 607 is fatal because it is a system directory for this rank

I would be grateful for any help in repair,

Dan

On 10/25/2017 04:17 PM, Daniel Davidson wrote:
A bit more news: I made the ceph-0 mds shut down, started the mds on ceph-1 and then told it the mds was repaired.  Everything ran great for about 5 hours and now it has crashed again.  Same error:

2017-10-25 15:13:07.344093 mon.0 [INF] fsmap e121828: 1/1/1 up {0=ceph-1=up:active} 2017-10-25 15:13:07.383445 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-25 15:13:07.480785 mon.0 [INF] osdmap e35296: 32 osds: 32 up, 32 in 2017-10-25 15:13:07.530337 mon.0 [INF] pgmap v28449919: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 5894 kB/s rd, 26 op/s 2017-10-25 15:13:08.473363 mon.0 [INF] mds.0 172.16.31.2:6802/3109594408 down:damaged
2017-10-25 15:13:08.473487 mon.0 [INF] fsmap e121829: 0/1/1 up, 1 damaged

If I:
#  rados -p igbhome_data rmomapkey 100.00000000 stray7_head
# ceph mds repaired 0

then I get:
2017-10-25 16:11:52.219916 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7) 2017-10-25 16:11:52.307975 mon.0 [INF] osdmap e35322: 32 osds: 32 up, 32 in 2017-10-25 16:11:52.357904 mon.0 [INF] pgmap v28450567: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 11773 kB/s rd, 5262 kB/s wr, 26 op/s 2017-10-25 16:11:53.325331 mon.0 [INF] mds.0 172.16.31.2:6802/2716803172 down:damaged
2017-10-25 16:11:53.325424 mon.0 [INF] fsmap e121882: 0/1/1 up, 1 damaged
2017-10-25 16:11:53.475087 mon.0 [INF] pgmap v28450568: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 39677 kB/s rd, 47236 B/s wr, 54 op/s 2017-10-25 16:11:54.590232 mon.0 [INF] pgmap v28450569: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 28105 kB/s rd, 3786 kB/s wr, 43 op/s 2017-10-25 16:11:55.719476 mon.0 [INF] pgmap v28450570: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 26284 kB/s rd, 3678 kB/s wr, 357 op/s 2017-10-25 16:11:56.830623 mon.0 [INF] pgmap v28450571: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 37249 kB/s rd, 5476 B/s wr, 358 op/s 2017-10-25 16:11:57.965330 mon.0 [INF] pgmap v28450572: 1536 pgs: 1536 active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 60769 kB/s rd, 53485 B/s wr, 41 op/s 2017-10-25 16:11:58.033787 mon.0 [INF] mds.? 172.16.31.2:6802/2725942008 up:boot 2017-10-25 16:11:58.033876 mon.0 [INF] fsmap e121883: 0/1/1 up, 1 up:standby, 1 damaged

Dan

On 10/25/2017 11:30 AM, Daniel Davidson wrote:
The system is down again saying it is missing the same stray7 again.

2017-10-25 11:24:29.736774 mds.0 [WRN] failed to reconnect caps for missing inodes:
2017-10-25 11:24:29.736779 mds.0 [WRN]  ino 100147160e6
2017-10-25 11:24:29.753665 mds.0 [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7)


Dan

On 10/25/2017 08:54 AM, Daniel Davidson wrote:
Thanks for the information.

I did:
# ceph daemon mds.ceph-0 scrub_path / repair recursive

Saw in the logs it finished

# ceph daemon mds.ceph-0 flush journal

Saw in the logs it finished

#ceph mds fail 0
#ceph mds repaired 0

And it went back to missing stray7 again.  I added that back like we did earlier and the system is back on line again, but the metadata errors still exist.

Dan

On 10/25/2017 07:50 AM, John Spray wrote:
Commands that start "ceph daemon" take mds.<name> rather than a rank
(notes on terminology here:
http://docs.ceph.com/docs/master/cephfs/standby/).  The name is how
you would refer to the daemon from systemd, it's often set to the
hostname where the daemon is running by default.

John

On Wed, Oct 25, 2017 at 2:30 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
I do have a problem with running the commands you mentioned to repair the
mds:

# ceph daemon mds.0 scrub_path
admin_socket: exception getting command descriptions: [Errno 2] No such file
or directory
admin_socket: exception getting command descriptions: [Errno 2] No such file
or directory

Any idea why that is not working?

Dan



On 10/25/2017 06:45 AM, Daniel Davidson wrote:
John, thank you so much.  After doing the initial rados command you
mentioned it is back up and running.  It did complain about a bunch of files which frankly are not important having duplicate inodes, but I will run those repair and scrub commands you mentioned and get it back clean again.

Dan

On 10/25/2017 03:55 AM, John Spray wrote:
On Tue, Oct 24, 2017 at 7:14 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
Our ceph system is having a problem.

A few days a go we had a pg that was marked as inconsistent, and today I
fixed it with a:

#ceph pg repair 1.37c

then a file was stuck as missing so I did a:

#ceph pg 1.37c mark_unfound_lost delete
pg has 1 objects unfound and apparently lost marking
OK, so "fixed" might be a bit of an overstatement here: while the PG is considered healthy, from CephFS's point of view what happened was
that some of its metadata just got blown away.

There are some (most) objects that CephFS can do without (it will just EIO when you try to read that file/dir), but there are some that are essential and will cause a whole MDS rank to be damaged (unstartable)
-- that's what's happened in your case.

That fixed the unfound file problem and all the pgs went active+clean.
A
few minutes later though, the FS seemed to pause and the MDS started
giving
errors.

# ceph -w
      cluster 7bffce86-9d7b-4bdf-a9c9-67670e68ca77
       health HEALTH_ERR
              mds rank 0 is damaged
              mds cluster is degraded
              noscrub,nodeep-scrub flag(s) set
       monmap e3: 4 mons at

{ceph-0=172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0}
              election epoch 652, quorum 0,1,2,3
ceph-0,ceph-1,ceph-2,ceph-3
        fsmap e121409: 0/1/1 up, 4 up:standby, 1 damaged
       osdmap e35220: 32 osds: 32 up, 32 in
              flags noscrub,nodeep-scrub,sortbitwise,require_jewel_osds         pgmap v28398840: 1536 pgs, 2 pools, 795 TB data, 329 Mobjects
              1595 TB used, 1024 TB / 2619 TB avail
                  1536 active+clean

Looking into the logs when I try a:

#ceph mds repaired 0

2017-10-24 12:01:27.354271 mds.0 172.16.31.3:6801/1949050374 75 :
cluster
[ERR] dir 607 object missing on disk; some files may be lost
(~mds0/stray7)

Any ideas as for what to do next, I am stumped.
So if this is really the only missing object, then it's your lucky
day, you lost a stray directory which usually contain just deleted
files (can contain something more important if you've had hardlinks
where the original file was later deleted).

The MDS goes damaged if it has a reference to a stay directory, but
the directory object isn't found.  OTOH if there is no reference to
the stray directory, it will happily recreate it for you. So, you can
do this:
rados -p <your metadata pool> rmomapkey 100.00000000 stray7_head

...to prompt the MDS to recreate the stray directory (the arguments
there are the magic internal names for ~mds0/stray7).

Then, if that was the only damage, your MDS will come up after you run
"ceph mds repaired 0".

There will still be some inconsistency resulting from removing the
stray dir, and possibly also from the disaster recovery tools that
you've run since, so you'll want to do a "ceph daemon mds.<id>
scrub_path / repair recursive".  This will probably output a bunch of
messages to the cluster log about things that it is repairing. Then
do "ceph daemon mds.<id> flush journal" to flush out the repairs it
has made, and restart the MDS daemon one more time ("ceph mds fail
0").

John

Dan


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux