John, thank you so much. After doing the initial rados command you
mentioned it is back up and running. It did complain about a bunch of
files which frankly are not important having duplicate inodes, but I
will run those repair and scrub commands you mentioned and get it back
clean again.
Dan
On 10/25/2017 03:55 AM, John Spray wrote:
On Tue, Oct 24, 2017 at 7:14 PM, Daniel Davidson
<danield@xxxxxxxxxxxxxxxx> wrote:
Our ceph system is having a problem.
A few days a go we had a pg that was marked as inconsistent, and today I
fixed it with a:
#ceph pg repair 1.37c
then a file was stuck as missing so I did a:
#ceph pg 1.37c mark_unfound_lost delete
pg has 1 objects unfound and apparently lost marking
OK, so "fixed" might be a bit of an overstatement here: while the PG
is considered healthy, from CephFS's point of view what happened was
that some of its metadata just got blown away.
There are some (most) objects that CephFS can do without (it will just
EIO when you try to read that file/dir), but there are some that are
essential and will cause a whole MDS rank to be damaged (unstartable)
-- that's what's happened in your case.
That fixed the unfound file problem and all the pgs went active+clean. A
few minutes later though, the FS seemed to pause and the MDS started giving
errors.
# ceph -w
cluster 7bffce86-9d7b-4bdf-a9c9-67670e68ca77
health HEALTH_ERR
mds rank 0 is damaged
mds cluster is degraded
noscrub,nodeep-scrub flag(s) set
monmap e3: 4 mons at
{ceph-0=172.16.31.1:6789/0,ceph-1=172.16.31.2:6789/0,ceph-2=172.16.31.3:6789/0,ceph-3=172.16.31.4:6789/0}
election epoch 652, quorum 0,1,2,3 ceph-0,ceph-1,ceph-2,ceph-3
fsmap e121409: 0/1/1 up, 4 up:standby, 1 damaged
osdmap e35220: 32 osds: 32 up, 32 in
flags noscrub,nodeep-scrub,sortbitwise,require_jewel_osds
pgmap v28398840: 1536 pgs, 2 pools, 795 TB data, 329 Mobjects
1595 TB used, 1024 TB / 2619 TB avail
1536 active+clean
Looking into the logs when I try a:
#ceph mds repaired 0
2017-10-24 12:01:27.354271 mds.0 172.16.31.3:6801/1949050374 75 : cluster
[ERR] dir 607 object missing on disk; some files may be lost (~mds0/stray7)
Any ideas as for what to do next, I am stumped.
So if this is really the only missing object, then it's your lucky
day, you lost a stray directory which usually contain just deleted
files (can contain something more important if you've had hardlinks
where the original file was later deleted).
The MDS goes damaged if it has a reference to a stay directory, but
the directory object isn't found. OTOH if there is no reference to
the stray directory, it will happily recreate it for you. So, you can
do this:
rados -p <your metadata pool> rmomapkey 100.00000000 stray7_head
...to prompt the MDS to recreate the stray directory (the arguments
there are the magic internal names for ~mds0/stray7).
Then, if that was the only damage, your MDS will come up after you run
"ceph mds repaired 0".
There will still be some inconsistency resulting from removing the
stray dir, and possibly also from the disaster recovery tools that
you've run since, so you'll want to do a "ceph daemon mds.<id>
scrub_path / repair recursive". This will probably output a bunch of
messages to the cluster log about things that it is repairing. Then
do "ceph daemon mds.<id> flush journal" to flush out the repairs it
has made, and restart the MDS daemon one more time ("ceph mds fail
0").
John
Dan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
