Re: MDS damaged

Daniel Davidson <danield@xxxxxxxxxxxxxxxx> · Wed, 25 Oct 2017 16:17:43 -0500

A bit more news: I made the ceph-0 mds shut down, started the mds on 
ceph-1 and then told it the mds was repaired.  Everything ran great for 
about 5 hours and now it has crashed again.  Same error:

2017-10-25 15:13:07.344093 mon.0 [INF] fsmap e121828: 1/1/1 up 
{0=ceph-1=up:active}
2017-10-25 15:13:07.383445 mds.0 [ERR] dir 607 object missing on disk; 
some files may be lost (~mds0/stray7)
2017-10-25 15:13:07.480785 mon.0 [INF] osdmap e35296: 32 osds: 32 up, 32 in
2017-10-25 15:13:07.530337 mon.0 [INF] pgmap v28449919: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 5894 
kB/s rd, 26 op/s
2017-10-25 15:13:08.473363 mon.0 [INF] mds.0 172.16.31.2:6802/3109594408 
down:damaged
2017-10-25 15:13:08.473487 mon.0 [INF] fsmap e121829: 0/1/1 up, 1 damaged

If I:
#  rados -p igbhome_data rmomapkey 100.00000000 stray7_head
# ceph mds repaired 0

then I get:
2017-10-25 16:11:52.219916 mds.0 [ERR] dir 607 object missing on disk; 
some files may be lost (~mds0/stray7)
2017-10-25 16:11:52.307975 mon.0 [INF] osdmap e35322: 32 osds: 32 up, 32 in
2017-10-25 16:11:52.357904 mon.0 [INF] pgmap v28450567: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 11773 
kB/s rd, 5262 kB/s wr, 26 op/s
2017-10-25 16:11:53.325331 mon.0 [INF] mds.0 172.16.31.2:6802/2716803172 
down:damaged
2017-10-25 16:11:53.325424 mon.0 [INF] fsmap e121882: 0/1/1 up, 1 damaged
2017-10-25 16:11:53.475087 mon.0 [INF] pgmap v28450568: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 39677 
kB/s rd, 47236 B/s wr, 54 op/s
2017-10-25 16:11:54.590232 mon.0 [INF] pgmap v28450569: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 28105 
kB/s rd, 3786 kB/s wr, 43 op/s
2017-10-25 16:11:55.719476 mon.0 [INF] pgmap v28450570: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 26284 
kB/s rd, 3678 kB/s wr, 357 op/s
2017-10-25 16:11:56.830623 mon.0 [INF] pgmap v28450571: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 37249 
kB/s rd, 5476 B/s wr, 358 op/s
2017-10-25 16:11:57.965330 mon.0 [INF] pgmap v28450572: 1536 pgs: 1536 
active+clean; 793 TB data, 1592 TB used, 1026 TB / 2619 TB avail; 60769 
kB/s rd, 53485 B/s wr, 41 op/s
2017-10-25 16:11:58.033787 mon.0 [INF] mds.? 172.16.31.2:6802/2725942008 
up:boot
2017-10-25 16:11:58.033876 mon.0 [INF] fsmap e121883: 0/1/1 up, 1 
up:standby, 1 damaged

Dan

On 10/25/2017 11:30 AM, Daniel Davidson wrote:
The system is down again saying it is missing the same stray7 again.

2017-10-25 11:24:29.736774 mds.0 [WRN] failed to reconnect caps for 
missing inodes:
2017-10-25 11:24:29.736779 mds.0 [WRN]  ino 100147160e6
2017-10-25 11:24:29.753665 mds.0 [ERR] dir 607 object missing on disk; 
some files may be lost (~mds0/stray7)

Dan

On 10/25/2017 08:54 AM, Daniel Davidson wrote:
Thanks for the information.

I did:
# ceph daemon mds.ceph-0 scrub_path / repair recursive

Saw in the logs it finished

# ceph daemon mds.ceph-0 flush journal

Saw in the logs it finished

#ceph mds fail 0
#ceph mds repaired 0

And it went back to missing stray7 again.  I added that back like we 
did earlier and the system is back on line again, but the metadata 
errors still exist.

Dan

On 10/25/2017 07:50 AM, John Spray wrote:
Commands that start "ceph daemon" take mds.<name> rather than a rank
(notes on terminology here:
http://docs.ceph.com/docs/master/cephfs/standby/).  The name is how
you would refer to the daemon from systemd, it's often set to the
hostname where the daemon is running by default.

John

On Wed, Oct 25, 2017 at 2:30 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
I do have a problem with running the commands you mentioned to 
repair the
mds:

# ceph daemon mds.0 scrub_path
admin_socket: exception getting command descriptions: [Errno 2] No 
such file
or directory
admin_socket: exception getting command descriptions: [Errno 2] No 
such file
or directory

Any idea why that is not working?

Dan

On 10/25/2017 06:45 AM, Daniel Davidson wrote:
John, thank you so much.  After doing the initial rados command you
mentioned it is back up and running.  It did complain about a 
bunch of files
which frankly are not important having duplicate inodes, but I 
will run
those repair and scrub commands you mentioned and get it back 
clean again.

Dan

On 10/25/2017 03:55 AM, John Spray wrote:
On Tue, Oct 24, 2017 at 7:14 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
Our ceph system is having a problem.

A few days a go we had a pg that was marked as inconsistent, and 
today I
fixed it with a:

#ceph pg repair 1.37c

then a file was stuck as missing so I did a:

#ceph pg 1.37c mark_unfound_lost delete
pg has 1 objects unfound and apparently lost marking
OK, so "fixed" might be a bit of an overstatement here: while the PG
is considered healthy, from CephFS's point of view what happened was
that some of its metadata just got blown away.

There are some (most) objects that CephFS can do without (it will 
just
EIO when you try to read that file/dir), but there are some that are
essential and will cause a whole MDS rank to be damaged 
(unstartable)
-- that's what's happened in your case.

That fixed the unfound file problem and all the pgs went 
active+clean.
A
few minutes later though, the FS seemed to pause and the MDS 
started
giving
errors.

# ceph -w
      cluster 7bffce86-9d7b-4bdf-a9c9-67670e68ca77
       health HEALTH_ERR
              mds rank 0 is damaged
              mds cluster is degraded
              noscrub,nodeep-scrub flag(s) set
       monmap e3: 4 mons at

{ceph-0=172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0} 

              election epoch 652, quorum 0,1,2,3
ceph-0,ceph-1,ceph-2,ceph-3
        fsmap e121409: 0/1/1 up, 4 up:standby, 1 damaged
       osdmap e35220: 32 osds: 32 up, 32 in
              flags 
noscrub,nodeep-scrub,sortbitwise,require_jewel_osds
        pgmap v28398840: 1536 pgs, 2 pools, 795 TB data, 329 
Mobjects
              1595 TB used, 1024 TB / 2619 TB avail
                  1536 active+clean

Looking into the logs when I try a:

#ceph mds repaired 0

2017-10-24 12:01:27.354271 mds.0 172.16.31.3:6801/1949050374 75 :
cluster
[ERR] dir 607 object missing on disk; some files may be lost
(~mds0/stray7)

Any ideas as for what to do next, I am stumped.
So if this is really the only missing object, then it's your lucky
day, you lost a stray directory which usually contain just deleted
files (can contain something more important if you've had hardlinks
where the original file was later deleted).

The MDS goes damaged if it has a reference to a stay directory, but
the directory object isn't found.  OTOH if there is no reference to
the stray directory, it will happily recreate it for you. So, you 
can
do this:
rados -p <your metadata pool> rmomapkey 100.00000000 stray7_head

...to prompt the MDS to recreate the stray directory (the arguments
there are the magic internal names for ~mds0/stray7).

Then, if that was the only damage, your MDS will come up after 
you run
"ceph mds repaired 0".

There will still be some inconsistency resulting from removing the
stray dir, and possibly also from the disaster recovery tools that
you've run since, so you'll want to do a "ceph daemon mds.<id>
scrub_path / repair recursive".  This will probably output a 
bunch of
messages to the cluster log about things that it is repairing. Then
do "ceph daemon mds.<id> flush journal" to flush out the repairs it
has made, and restart the MDS daemon one more time ("ceph mds fail
0").

John

Dan

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com