Re: MDS cache always increasing

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Ow it got worse after the upgrade to Reef (was running Quincy). With Quincy the memory usage was also a lot of times around 95% and some swap usage, but never exceeding both to the point of crashing. 

Kind regards, 
Sake
> Op 31-08-2024 09:15 CEST schreef Alexander Patrakov <patrakov@xxxxxxxxx>:
> 
>  
> Got it.
> 
> However, to narrow down the issue, I suggest that you test whether it
> still exists after the following changes:
> 
> 1. Reduce max_mds to 1.
> 2. Do not reduce max_mds to 1, but migrate all clients from a direct
> CephFS mount to NFS.
> 
> On Sat, Aug 31, 2024 at 2:55 PM Sake Ceph <ceph@xxxxxxxxxxx> wrote:
> >
> > I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9.
> >
> > Kind regards,
> > Sake
> >
> > > Op 31-08-2024 08:34 CEST schreef Alexander Patrakov <patrakov@xxxxxxxxx>:
> > >
> > >
> > > Hello Sake,
> > >
> > > The combination of two active MDSs and RHEL8 does ring a bell, and I
> > > have seen this with Quincy, too. However, what's relevant is the
> > > kernel version on the clients. If they run the default 4.18.x kernel
> > > from RHEL8, please either upgrade to the mainline kernel or decrease
> > > max_mds to 1. If they run a modern kernel, then it is something I do
> > > not know about.
> > >
> > > On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph <ceph@xxxxxxxxxxx> wrote:
> > > >
> > > > @Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem.
> > > >
> > > > @Alexander: the problem is in the active nodes, the standby replay don't have issues anymore.
> > > >
> > > > Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%.
> > > > It looks to me the MDS server is caching something forever while it isn't being used..
> > > >
> > > > The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts.
> > > >
> > > > Kind regards,
> > > > Sake
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> > >
> > >
> > >
> > > --
> > > Alexander Patrakov
> 
> 
> 
> -- 
> Alexander Patrakov
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux