Re: cephfs kernel client instability

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Ilya/Kjetil,

I've done some debugging and tcpdump-ing to see what the interaction between the kernel client and the mon looks like.  Indeed - CEPH_MSG_MAX_FRONT defined as 16Mb seems low for the default mon messages for our cluster (with osd_mon_messages_max at 100).  We have about 3500 osd's, and the kernel advertises itself as older than Luminous, so it gets full map updates.  The FRONT message size on the wire I saw was over 24Mb.  I'll try setting osd_mon_messages_max to 30 and do some more testing, but from the debugging it definitely seems like the issue.

Is the kernel driver really not up to date to be considered at least a Luminous client by the mon (i.e. it has some feature really missing)?  I looked at the bits, and the MON seems to want is bit 59 in ceph features shared by FS_BTIME, FS_CHANGE_ATTR, MSG_ADDR2.  Can the kernel client be used when setting require-min-compat to luminous (either with the 4.19.x kernel or the Redhat/Centos 7.6 kernel)?  Some background here would be helpful.

Thanks for your help and suggestions!

Andras


On 1/16/19 5:20 AM, Ilya Dryomov wrote:
On Wed, Jan 16, 2019 at 1:27 AM Kjetil Joergensen <kjetil@xxxxxxxxxxxx> wrote:
Hi,

you could try reducing "osd map message max", some code paths that end up as -EIO (kernel: libceph: mon1 *** io error) is exceeding include/linux/ceph/libceph.h:CEPH_MSG_MAX_{FRONT,MIDDLE,DATA}_LEN.

This "worked for us" - YMMV.
Kjetil, how big is your cluster?  Do you remember the circumstances
of when you started seeing these errors?

Andras, please let us know if this resolves the issue.  Decreasing
"osd map message max" for large clusters can help with the overall
memory consumption and is probably a good idea in general, but then
these kernel client limits are pretty arbitrary, so we can look at
bumping them.

Thanks,

                 Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux