Re: cephfs kernel client instability

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Ilya,

Thanks for the clarification - very helpful.
I've lowered osd_map_messages_max to 10, and this resolves the issue about the kernel being unhappy about large messages when the OSDMap changes.  One comment here though: you mentioned that Luminous uses 40 as the default, which is indeed the case.  The documentation for Luminous (and master), however, says that the default is 100.

One other follow-up question on the kernel client about something I've been seeing while testing.  Does the kernel client clean up when the MDS asks due to cache pressure?  On a machine I ran something that touches a lot of files, so the kernel client accumulated over 4 million caps.  Many hours after all the activity finished (i.e. many hours after anything accesses ceph on that node) the kernel client still holds millions of caps, and the MDS periodically complains about clients not responding to cache pressure.  How is this supposed to be handled?  Obviously asking the kernel to drop caches via /proc/sys/vm/drop_caches does a very thorough cleanup, but something in the middle would be better.

Andras


On 1/16/19 1:45 PM, Ilya Dryomov wrote:
On Wed, Jan 16, 2019 at 7:12 PM Andras Pataki
<apataki@xxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Ilya/Kjetil,

I've done some debugging and tcpdump-ing to see what the interaction
between the kernel client and the mon looks like.  Indeed -
CEPH_MSG_MAX_FRONT defined as 16Mb seems low for the default mon
messages for our cluster (with osd_mon_messages_max at 100).  We have
about 3500 osd's, and the kernel advertises itself as older than
This is too big, especially for a fairly large cluster such as yours.
The default was reduced to 40 in luminous.  Given about 3500 OSDs, you
might want to set it to 20 or even 10.

Luminous, so it gets full map updates.  The FRONT message size on the
wire I saw was over 24Mb.  I'll try setting osd_mon_messages_max to 30
and do some more testing, but from the debugging it definitely seems
like the issue.

Is the kernel driver really not up to date to be considered at least a
Luminous client by the mon (i.e. it has some feature really missing)?  I
looked at the bits, and the MON seems to want is bit 59 in ceph features
shared by FS_BTIME, FS_CHANGE_ATTR, MSG_ADDR2.  Can the kernel client be
used when setting require-min-compat to luminous (either with the 4.19.x
kernel or the Redhat/Centos 7.6 kernel)?  Some background here would be
helpful.
Yes, the kernel client is missing support for that feature bit, however
4.13+ and RHEL 7.5+ _can_ be used with require-min-compat-client set to
luminous.  See

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/027002.html

Thanks,

                 Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux