On Wed, Jul 4, 2018 at 7:02 PM Dennis Kramer (DBS) <dennis@xxxxxxxxx> wrote: > > Hi, > > I have managed to get cephfs mds online again...for a while. > > These topics covers more or less my symptoms and helped me get it up > and running again: > - https://www.spinics.net/lists/ceph-users/msg45696.h > tml > - http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-December/ > 023133.html > > After some time it all goes down again and keeps in a loop trying to > get into an "active" state then after a while it crashes again. > Logs from the MDS right before it crashes: > > 0> 2018-07-04 11:34:54.657595 7f50f1c29700 -1 /build/ceph- > 12.2.5/src/mds/MDCache.cc: In function 'void > > MDCache::add_inode(CInode*)' thread 7f50f1c29700 time 2018-07-04 > 11:34:54.638462 > > /build/ceph-12.2.5/src/mds/MDCache.cc: 262: FAILED assert(!p) > > Cluster logging: > 2018-07-04 12:50:04.741625 mds.mds01 [ERR] dir 0x1000098a246 object > missing on disk; some files may be lost (<obfuscated file path) > 2018-07-04 12:50:16.352824 mon.mon01 [ERR] MDS health message (mds.0): > Metadata damage detected > 2018-07-04 12:50:16.480045 mon.mon01 [ERR] Health check failed: 1 MDSs > report damaged metadata (MDS_DAMAGE) > 2018-07-04 12:53:36.194056 mds.mds01 [ERR] loaded dup inode > 0x10000989e52 [2,head] v1104251 at <file>, but inode 0x10000989e52.head > v37 already exists at <another file> > > CephFS won't stay up for long, after some time it crashes and I need to > reset the fs to get it back again. > > I'm at a loss here. I guess you did reset mds journal. have you run complete recovery sequence? cephfs-data-scan init cephfs-data-scan scan_extents <data pool> cephfs-data-scan scan_inodes <data pool> cephfs-data-scan scan_links see http://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ > > On Wed, 2018-06-27 at 21:38 +0800, Yan, Zheng wrote: > > On Wed, Jun 27, 2018 at 6:16 PM Dennis Kramer (DT) <dennis@xxxxxxxxx> > > wrote: > > > > > > > > > Hi, > > > > > > Currently i'm running Ceph Luminous 12.2.5. > > > > > > This morning I tried running Multi MDS with: > > > ceph fs set <fs_name> max_mds 2 > > > > > > I have 5 MDS servers. After running above command, > > > I had 2 active MDSs, 2 standby-active and 1 standby. > > > > > > And after trying a failover on one > > > of the active MDSs, a standby-active did a replay but crashed > > > (laggy or > > > crashed). Memory and CPU went sky high on the MDS and was > > > unresponsive > > > after some time. I ended up with the one active MDS but got stuck > > > with a > > > degraded filesystem and warning messages about MDS behind on > > > trimming. > > > > > > I never got any additional MDS active since then. I tried > > > restarting the > > > last active MDS (because the filesystem was becoming unresponsive > > > and had > > > a load of slow requets) and it never got passed replay -> resolve. > > > My MDS > > > cluster still isn't active... :( > > What is the 'ceph -w' ouput? If you have enabled multi-active mds. > > All > > mds ranks need to enter the resolve 'state' before they can continue > > to recover. > > > > > > > > > > > > > > > What is the "resolve" state? I have never seen that before pre- > > > Luminous. > > > Debug on 20 doesn't give me much. > > > > > > Also tried removing the Multi MDS setup, but my CephFS cluster > > > won't go > > > active. How can I get my CephFS up and running again in an active > > > state. > > > > > > Please help. > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |