Re: [Ceph-users] Re: MDS failing under load with large cache sizes

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, Jul 25, 2019 at 12:49 PM Janek Bevendorff
<janek.bevendorff@xxxxxxxxxxxxx> wrote:
>
>
> > Based on that message, it would appear you still have an inode limit
> > in place ("mds_cache_size"). Please unset that config option. Your
> > mds_cache_memory_limit is apparently ~19GB.
>
> No, I do not have an inode limit set. Only the memory limit.
>
>
> > There is another limit mds_max_caps_per_client (default 1M) which the
> > client is hitting. That's why the MDS is recalling caps from the
> > client and not because any cache memory limit is hit. It is not
> > recommend you increase this.
> Okay, this this setting isn't documented either and I did not change it,
> but it's also quite clear that it isn't working. My MDS hasn't crashed
> yet (without the recall settings it would have), but ceph fs status is
> reporting 14M inodes at this point and the number is slowly going up.

Can you share two captures of `ceph daemon mds.X perf dump` about 1
second apart.

You can also try increasing the aggressiveness of the MDS recall but
I'm surprised it's still a problem with the settings I gave you:

ceph config set mds mds_recall_max_caps 15000
ceph config set mds mds_recall_max_decay_rate 0.75

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux