Re: mds crash loop

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Nov 11, 2019 at 5:09 PM Karsten Nielsen <karsten@xxxxxxxxxx> wrote:
>
> I started a job that moved some files around in the cephfs cluster that resulted in the mds to go back into the crash loop.
> Logs are here:
> http://s3.foo-bar.dk/mds-dumps/mds.log-20191111
>
> Any help would be appriciated.
>

looks like snaptable is corrupted.

nautilus version (14.2.2) of ‘cephfs-data-scan scan_links’  can fix
snaptable. hopefully it will fix your issue.

you don't need to upgrade whole cluster. Just install nautilus in a
temp machine or compile ceph from source.



> - Karsten
>
> -----Original message-----
> From:   Yan, Zheng <ukernel@xxxxxxxxx>
> Sent:   Thu 07-11-2019 14:20
> Subject:        Re:  Re: mds crash loop
> To:     Karsten Nielsen <karsten@xxxxxxxxxx>;
> CC:     ceph-users@xxxxxxx;
> > On Thu, Nov 7, 2019 at 6:40 PM Karsten Nielsen <karsten@xxxxxxxxxx> wrote:
> > >
> > > That is awesome.
> > >
> > > Now I just need to figure out where the lost+found files needs to go.
> > > And what happened to the missing objects for the dirs.
> > >
> >
> > lost+found files are likely files that were deleted. you can keep the
> > lost+found dir for a while, then delete the 'lost+found' directory.
> >
> > for 'missing object' dirs, mv all of them to a temp directory, such as
> > /mnt/cephfs/missing_obj_dirs.
> > Then run command 'ceph daemon mds.x scrub_patch /missing_obj_dirs
> > force recursive repair'. wait a minute, the rm -rf
> > /mnt/cephfs/missing_obj_dirs
> >
> > > Any tool that is able to do that ?
> > >
> > > Thanks
> > > - Karsten
> > >
> > > -----Original message-----
> > > From:   Yan, Zheng <ukernel@xxxxxxxxx>
> > > Sent:   Thu 07-11-2019 09:22
> > > Subject:        Re:  Re: mds crash loop
> > > To:     Karsten Nielsen <karsten@xxxxxxxxxx>;
> > > CC:     ceph-users@xxxxxxx;
> > > > I have tracked down the root cause. See
> > https://tracker.ceph.com/issues/42675
> > > >
> > > > Regards
> > > > Yan, Zheng
> > > >
> > > > On Thu, Nov 7, 2019 at 4:01 PM Karsten Nielsen <karsten@xxxxxxxxxx> wrote:
> > > > >
> > > > > -----Original message-----
> > > > > From:   Yan, Zheng <ukernel@xxxxxxxxx>
> > > > > Sent:   Thu 07-11-2019 07:21
> > > > > Subject:        Re:  Re: mds crash loop
> > > > > To:     Karsten Nielsen <karsten@xxxxxxxxxx>;
> > > > > CC:     ceph-users@xxxxxxx;
> > > > > > On Thu, Nov 7, 2019 at 5:50 AM Karsten Nielsen <karsten@xxxxxxxxxx>
> > wrote:
> > > > > > >
> > > > > > > -----Original message-----
> > > > > > > From:   Yan, Zheng <ukernel@xxxxxxxxx>
> > > > > > > Sent:   Wed 06-11-2019 14:16
> > > > > > > Subject:        Re:  mds crash loop
> > > > > > > To:     Karsten Nielsen <karsten@xxxxxxxxxx>;
> > > > > > > CC:     ceph-users@xxxxxxx;
> > > > > > > > On Wed, Nov 6, 2019 at 4:42 PM Karsten Nielsen <karsten@xxxxxxxxxx>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > -----Original message-----
> > > > > > > > > From:   Yan, Zheng <ukernel@xxxxxxxxx>
> > > > > > > > > Sent:   Wed 06-11-2019 08:15
> > > > > > > > > Subject:        Re:  mds crash loop
> > > > > > > > > To:     Karsten Nielsen <karsten@xxxxxxxxxx>;
> > > > > > > > > CC:     ceph-users@xxxxxxx;
> > > > > > > > > > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen
> > <karsten@xxxxxxxxxx>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > Last week I upgraded my ceph cluster from luminus to mimic
> > 13.2.6
> > > > > > > > > > > It was running fine for a while but yesterday my mds went
> > into a
> > > > crash
> > > > > > > > loop.
> > > > > > > > > > >
> > > > > > > > > > > I have 1 active and 1 standby mds for my cephfs both of which
> > is
> > > > > > running
> > > > > > > > the
> > > > > > > > > > same crash loop.
> > > > > > > > > > > I am running ceph based on
> > https://hub.docker.com/r/ceph/daemon
> > > > > > version
> > > > > > > > > > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store.
> > > > > > > > > > >
> > > > > > > > > > > Log details are: https://paste.debian.net/1113943/
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > please try again with debug_mds=20.  Thanks
> > > > > > > > > >
> > > > > > > > > > Yan, Zheng
> > > > > > > > >
> > > > > > > > > Yes I have set that and had to move to pastebin.com as debian
> > > > apperently
> > > > > > only
> > > > > > > > supports 150k
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > https://pastebin.com/Gv7c5h54
> > > > > > > > >
> > > > > > > >
> > > > > > > > Looks like on-disk root inode is corrupted. have you encountered any
> > > > > > > > unusually things during the upgrade?
> > > > > > > >
> > > > > > > > please run 'rados -p <cephfs metadata pool> stat 1.00000000.inode' ,
> > > > > > > > check if the object is modified before or after the 'luminous ->
> > > > > > > > 13.2.6' upgrade.
> > > > > > > > To fix the corrupted object.  Run  'cephfs-data-scan init
> > > > > > > > --force-init'. Then restart mds. After mds become active, run 'ceph
> > > > > > > > daemon mds.x scrub_path / force repair'
> > > > > > > >
> > > > > > >
> > > > > > > I followed the steps I got the mds started but now a lot of files are
> > in
> > > > > > lost+found 24283 and I have these errors in the mds log
> > > > > > >
> > > > > >  'cephfs-data-scan init --force-init' does not move files into
> > > > > > lost+found. have you ever run other 'cephfs-data-scan foo' command or
> > > > > > 'cephfs-journal-tool foo' command?
> > > > >
> > > > > I have had a similar problem with the cluster before where I went through
> > the
> > > > cycle of:
> > > > > https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ -> Using an
> > > > alternate metadata pool for recovery
> > > > >
> > > > > I did run the cephfs-journal-tool journal reset command, mostly because
> > > > cephfs is not that utilized so I thought it was safe to do as after the
> > upgrade
> > > > the cluster has not been used much, so data lose would be minimal -
> > apparently
> > > > I was wrong.
> > > > >
> > > > > >
> > > > > > > 2019-11-06 20:20:18.215 7f0bd9090700  1 mds.0.32011 cluster recovered.
> > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700  0 mds.0.cache.dir(0x100013acfcb)
> > > > > > _fetched missing object for [dir 0x100013acfcb
> > > > /nextcloud/custom_apps/carnet/
> > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n()
> > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dc4f5100]
> > > > > > > 2019-11-06 20:20:19.019 7f0bd2dfa700 -1 log_channel(cluster) log
> > [ERR] :
> > > > dir
> > > > > > 0x100013acfcb object missing on disk; some files may be lost
> > > > > > (/nextcloud/custom_apps/carnet)
> > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700  0 mds.0.cache.dir(0x100013a3156)
> > > > > > _fetched missing object for [dir 0x100013a3156
> > /nextcloud/custom_apps/mail/
> > > > > > [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741888|fetching f() n()
> > > > > > hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55d4dcc40000]
> > > > > > > 2019-11-06 20:20:19.275 7f0bd2dfa700 -1 log_channel(cluster) log
> > [ERR] :
> > > > dir
> > > > > > 0x100013a3156 object missing on disk; some files may be lost
> > > > > > (/nextcloud/custom_apps/mail)
> > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700  0 mds.0.cache.dir(0x100013abb3c)
> > > > > > _fetched missing object for [dir 0x100013abb3c
> > > > > > /nextcloud/custom_apps/passwords/ [2,head] auth v=0 cv=0/0 ap=1+0+0
> > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1
> > > > > > 0x55d4dcc40700]
> > > > > > > 2019-11-06 20:20:19.371 7f0bd2dfa700 -1 log_channel(cluster) log
> > [ERR] :
> > > > dir
> > > > > > 0x100013abb3c object missing on disk; some files may be lost
> > > > > > (/nextcloud/custom_apps/passwords)
> > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700  0 mds.0.cache.dir(0x100013a9b9b)
> > > > > > _fetched missing object for [dir 0x100013a9b9b
> > > > > > /nextcloud/custom_apps/phonetrack/ [2,head] auth v=0 cv=0/0 ap=1+0+0
> > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1
> > > > > > 0x55d4dcc40e00]
> > > > > > > 2019-11-06 20:20:19.383 7f0bd2dfa700 -1 log_channel(cluster) log
> > [ERR] :
> > > > dir
> > > > > > 0x100013a9b9b object missing on disk; some files may be lost
> > > > > > (/nextcloud/custom_apps/phonetrack)
> > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700  0 mds.0.cache.dir(0x100013a2659)
> > > > > > _fetched missing object for [dir 0x100013a2659
> > > > > > /nextcloud/custom_apps/richdocuments/ [2,head] auth v=0 cv=0/0 ap=1+0+0
> > > > > > state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1
> > > > > > 0x55d4dcc41500]
> > > > > > > 2019-11-06 20:20:19.431 7f0bd2dfa700 -1 log_channel(cluster) log
> > [ERR] :
> > > > dir
> > > > > > 0x100013a2659 object missing on disk; some files may be lost
> > > > > > (/nextcloud/custom_apps/richdocuments)
> > > > > > > 2019-11-06 20:20:22.360 7f0bd9090700  1 mds.k8s-node-01 Updating MDS
> > map
> > > > to
> > > > > > version 32015 from mon.1
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > - Karsten
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Thanks for any hints
> > > > > > > > > > > - Karsten
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > > > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > _______________________________________________
> > > > > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > > >
> > > >
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> >
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux