On Tue, Oct 24, 2017 at 7:14 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
> Our ceph system is having a problem.
>
> A few days a go we had a pg that was marked as inconsistent, and today I
> fixed it with a:
>
> #ceph pg repair 1.37c
>
> then a file was stuck as missing so I did a:
>
> #ceph pg 1.37c mark_unfound_lost delete
> pg has 1 objects unfound and apparently lost marking
OK, so "fixed" might be a bit of an overstatement here: while the PG
is considered healthy, from CephFS's point of view what happened was
that some of its metadata just got blown away.
There are some (most) objects that CephFS can do without (it will just
EIO when you try to read that file/dir), but there are some that are
essential and will cause a whole MDS rank to be damaged (unstartable)
-- that's what's happened in your case.
> That fixed the unfound file problem and all the pgs went active+clean. A
> few minutes later though, the FS seemed to pause and the MDS started giving
> errors.
>
> # ceph -w
> cluster 7bffce86-9d7b-4bdf-a9c9-67670e68ca77
> health HEALTH_ERR
> mds rank 0 is damaged
> mds cluster is degraded
> noscrub,nodeep-scrub flag(s) set
> monmap e3: 4 mons at
> {ceph-0=172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0}
> election epoch 652, quorum 0,1,2,3 ceph-0,ceph-1,ceph-2,ceph-3
> fsmap e121409: 0/1/1 up, 4 up:standby, 1 damaged
> osdmap e35220: 32 osds: 32 up, 32 in
> flags noscrub,nodeep-scrub,sortbitwise,require_jewel_osds
> pgmap v28398840: 1536 pgs, 2 pools, 795 TB data, 329 Mobjects
> 1595 TB used, 1024 TB / 2619 TB avail
> 1536 active+clean
>
> Looking into the logs when I try a:
>
> #ceph mds repaired 0
>
> 2017-10-24 12:01:27.354271 mds.0 172.16.31.3:6801/1949050374 75 : cluster
> [ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7)
>
> Any ideas as for what to do next, I am stumped.
So if this is really the only missing object, then it's your lucky
day, you lost a stray directory which usually contain just deleted
files (can contain something more important if you've had hardlinks
where the original file was later deleted).
The MDS goes damaged if it has a reference to a stay directory, but
the directory object isn't found. OTOH if there is no reference to
the stray directory, it will happily recreate it for you. So, you can
do this:
rados -p <your metadata pool> rmomapkey 100.00000000 stray7_head
...to prompt the MDS to recreate the stray directory (the arguments
there are the magic internal names for ~mds0/stray7).
Then, if that was the only damage, your MDS will come up after you run
"ceph mds repaired 0".
There will still be some inconsistency resulting from removing the
stray dir, and possibly also from the disaster recovery tools that
you've run since, so you'll want to do a "ceph daemon mds.<id>
scrub_path / repair recursive". This will probably output a bunch of
messages to the cluster log about things that it is repairing. Then
do "ceph daemon mds.<id> flush journal" to flush out the repairs it
has made, and restart the MDS daemon one more time ("ceph mds fail
0").
John
>
> Dan
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
