Hi,
I've tried to rm the mds0.openfiles and did a `ceph config set mds
mds_oft_prefetch_dirfrags false` but still with the same result of ceph
status reporting the daemon as up and not a lot more.
I also tried setting the cache to ridiculously small(128M) but the MDS'
memory usage would still go up to 50+GB either way.
cluster:
health: HEALTH_WARN
1 MDSs report oversized cache
[...]
services
[...]
mds: 1/1 daemons up
Cheers,
Izzy
On Fri, 11 Feb 2022 at 09:32, Dan van der Ster <dvanders@xxxxxxxxx> wrote:
> Hi,
>
> Is the memory ballooning while the MDS is active or could it be while
> it is rejoining the cluster?
> If the latter, this could be another case of:
> https://tracker.ceph.com/issues/54253
>
> Cheers, Dan
>
>
> On Wed, Feb 9, 2022 at 7:23 PM Izzy Kulbe <ceph@xxxxxxx> wrote:
> >
> > Hi,
> >
> > last weekend we upgraded one of our clusters from 16.2.5 to 16.2.7 using
> > cephadm.
> >
> > The upgrade itself seemed to run without a problem but shortly after the
> > upgrade we noticed the servers holding the MDS containers being laggy,
> then
> > unresponsive, then crashing outright due getting reaped by the OOM
> killer.
> > The MDS would restart, at some point resulting in a reset of the whole
> > machine.
> >
> > At this point the cluster status started showing a lot of PGs with
> > snaptrim/snaptrim_wait status, so we waited for those to finish with the
> > MDS shutdown.
> >
> > So far everything we've tried changed nothing about our situation - our
> MDS
> > will try to hog extreme amounts of memory they never needed before. Even
> > when we push the cache down to 1G or reset it to the default the will
> > report as having oversized caches, using up 150+GB of RAM+swap until
> > resource exhaustion makes the OOM killer reap them again.
> >
> > Looking at the output of ceph fs status and the logs it seems it's trying
> > to push too much into the cache. The MDS crashes at around 25-30M inodes
> > and a few thousand dirs. ceph fs status reports the MDS as being active,
> > unlike a few other posts with similar issues the MDS does not seem to be
> > stuck in replay. The journal reports no issues. Additionally perf reports
> > no strays, nor does listomapvals.
> >
> > The cluster contains ~33TB in ~220M files. There are multiple snapshots
> for
> > several folders. While ceph fs status reports the MDS as active, I am not
> > able to mount the filesystem.
> >
> > I will probably recreate/resync this soon but I'd still like to find out
> > why this happened since we plan on using Ceph in a few other applications
> > but at the moment I wouldn't feel comfortable using CephFS given that a
> > single cephadm upgrade command made these amounts of data unavailable.
> >
> > Below are a few lines of log that basically make up the MDS' logs until
> it
> > runs into "mds.0.purge_queue push: pushing inode" and shortly after that
> > the crash.
> >
> > Any pointers to what went wrong here, how to debug this further or how to
> > fix this would be greatly appreciated. I'll probably have the broken
> system
> > for another day or two before resetting/recreating the FS.
> >
> > Thanks,
> > Izzy Kulbe
> >
> >
> > MDS Log:
> >
> > 20 mds.0.cache.dir(0x604.001111000*) lookup_exact_snap (head,
> '10000eff917')
> > 10 mds.0.cache.snaprealm(0x10000eff917 seq 3398 0x55c541856200)
> > adjust_parent 0 -> 0x55ba9f1e2e00
> > 12 mds.0.cache.dir(0x604.001111000*) add_primary_dentry [dentry
> > #0x100/stray4/10000eff917 [d89,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000eff917 state=1073741824 0x55c541854a00]
> > 12 mds.0.cache.dir(0x604.001111000*) _fetched got [dentry
> > #0x100/stray4/10000eff917 [d89,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000eff917 state=1073741824 0x55c541854a00] [inode 0x10000eff917
> > [...d89,head] ~mds0/stray4/10000eff917/ auth v38642807
> > snaprealm=0x55c541856200 f(v0 m2021-12-29T05:05:25.519523+0000) n(v4
> > rc2021-12-29T05:05:25.527523+0000 1=0+1) old_inodes=1 (iversion lock)
> > 0x55c541851180]
> > 15 mds.0.cache.ino(0x10000eff917) maybe_ephemeral_rand unlinked
> directory:
> > cannot ephemeral random pin [inode 0x10000eff917 [...d89,head]
> > ~mds0/stray4/10000eff917/ auth v38642807 snaprealm=0x55c541856200 f(v0
> > m2021-12-29T05:05:25.519523+0000) n(v4 rc2021-12-29T05:05:25.527523+0000
> > 1=0+1) old_inodes=1 (iversion lock) 0x55c541851180]
> > 20 mds.0.cache.dir(0x604.001111000*) _fetched pos 137 marker 'i' dname
> > '10000efded1 [daf,head]
> > 20 mds.0.cache.dir(0x604.001111000*) lookup (head, '10000efded1')
> > 20 mds.0.cache.dir(0x604.001111000*) miss -> (1001124a935,head)
> > 20 mds.0.cache.ino(0x10000efded1) decode_snap_blob
> snaprealm(0x10000efded1
> > seq d9e lc 0 cr d9e cps daf snaps={}
> > past_parent_snaps=be5,c3d,c95,ced,d45,d9d 0x55c541856400)
> > 20 mds.0.cache.dir(0x604.001111000*) lookup_exact_snap (head,
> '10000efded1')
> > 10 mds.0.cache.snaprealm(0x10000efded1 seq 3486 0x55c541856400)
> > adjust_parent 0 -> 0x55ba9f1e2e00
> > 12 mds.0.cache.dir(0x604.001111000*) add_primary_dentry [dentry
> > #0x100/stray4/10000efded1 [daf,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000efded1 state=1073741824 0x55c541854f00]
> > 12 mds.0.cache.dir(0x604.001111000*) _fetched got [dentry
> > #0x100/stray4/10000efded1 [daf,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000efded1 state=1073741824 0x55c541854f00] [inode 0x10000efded1
> > [...daf,head] ~mds0/stray4/10000efded1/ auth v38652010
> > snaprealm=0x55c541856400 f(v0 m2022-01-04T19:09:57.033258+0000) n(v118
> > rc2022-01-04T19:09:57.041258+0000 1=0+1) old_inodes=6 (iversion lock)
> > 0x55c541851700]
> > 15 mds.0.cache.ino(0x10000efded1) maybe_ephemeral_rand unlinked
> directory:
> > cannot ephemeral random pin [inode 0x10000efded1 [...daf,head]
> > ~mds0/stray4/10000efded1/ auth v38652010 snaprealm=0x55c541856400 f(v0
> > m2022-01-04T19:09:57.033258+0000) n(v118
> rc2022-01-04T19:09:57.041258+0000
> > 1=0+1) old_inodes=6 (iversion lock) 0x55c541851700]
> > 20 mds.0.cache.dir(0x604.001111000*) _fetched pos 136 marker 'i' dname
> > '10000efad6f [daf,head]
> > 20 mds.0.cache.dir(0x604.001111000*) lookup (head, '10000efad6f')
> > 20 mds.0.cache.dir(0x604.001111000*) miss -> (1000539afa8,head)
> > 20 mds.0.cache.ino(0x10000efad6f) decode_snap_blob
> snaprealm(0x10000efad6f
> > seq d9e lc 0 cr d9e cps daf snaps={}
> > past_parent_snaps=be5,c3d,c95,ced,d45,d9d 0x55c541856600)
> > 20 mds.0.cache.dir(0x604.001111000*) lookup_exact_snap (head,
> '10000efad6f')
> > 10 mds.0.cache.snaprealm(0x10000efad6f seq 3486 0x55c541856600)
> > adjust_parent 0 -> 0x55ba9f1e2e00
> > 12 mds.0.cache.dir(0x604.001111000*) add_primary_dentry [dentry
> > #0x100/stray4/10000efad6f [daf,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000efad6f state=1073741824 0x55c541855400]
> > 12 mds.0.cache.dir(0x604.001111000*) _fetched got [dentry
> > #0x100/stray4/10000efad6f [daf,head] auth (dversion lock) pv=0 v=38664524
> > ino=0x10000efad6f state=1073741824 0x55c541855400] [inode 0x10000efad6f
> > [...daf,head] ~mds0/stray4/10000efad6f/ auth v38652024
> > snaprealm=0x55c541856600 f() n(v0 rc2022-01-04T19:10:53.334202+0000
> 1=0+1)
> > old_inodes=6 (iversion lock) 0x55c54185c000]
> > 15 mds.0.cache.ino(0x10000efad6f) maybe_ephemeral_rand unlinked
> directory:
> > cannot ephemeral random pin [inode 0x10000efad6f [...daf,head]
> > ~mds0/stray4/10000efad6f/ auth v38652024 snaprealm=0x55c541856600 f()
> n(v0
> > rc2022-01-04T19:10:53.334202+0000 1=0+1) old_inodes=6 (iversion lock)
> > 0x55c54185c000]
> > 20 mds.0.cache.dir(0x604.001111000*) _fetched pos 135 marker 'i' dname
> > '10000ef98c9 [d89,head]
> > 20 mds.0.cache.dir(0x604.001111000*) lookup (head, '10000ef98c9')
> > 20 mds.0.cache.dir(0x604.001111000*) miss -> (100113dd761,head)
> > 20 mds.0.cache.ino(0x10000ef98c9) decode_snap_blob
> snaprealm(0x10000ef98c9
> > seq d46 lc 0 cr d46 cps d89 snaps={}
> > past_parent_snaps=b8b,be5,c3d,c95,ced,d45 0x55c541856800)
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
