Hi Ilya/Kjetil,
I've done some debugging and tcpdump-ing to see what the interaction
between the kernel client and the mon looks like. Indeed -
CEPH_MSG_MAX_FRONT defined as 16Mb seems low for the default mon
messages for our cluster (with osd_mon_messages_max at 100). We have
about 3500 osd's, and the kernel advertises itself as older than
Luminous, so it gets full map updates. The FRONT message size on the
wire I saw was over 24Mb. I'll try setting osd_mon_messages_max to 30
and do some more testing, but from the debugging it definitely seems
like the issue.
Is the kernel driver really not up to date to be considered at least a
Luminous client by the mon (i.e. it has some feature really missing)? I
looked at the bits, and the MON seems to want is bit 59 in ceph features
shared by FS_BTIME, FS_CHANGE_ATTR, MSG_ADDR2. Can the kernel client be
used when setting require-min-compat to luminous (either with the 4.19.x
kernel or the Redhat/Centos 7.6 kernel)? Some background here would be
helpful.
Thanks for your help and suggestions!
Andras
On 1/16/19 5:20 AM, Ilya Dryomov wrote:
On Wed, Jan 16, 2019 at 1:27 AM Kjetil Joergensen <kjetil@xxxxxxxxxxxx> wrote:
Hi,
you could try reducing "osd map message max", some code paths that end up as -EIO (kernel: libceph: mon1 *** io error) is exceeding include/linux/ceph/libceph.h:CEPH_MSG_MAX_{FRONT,MIDDLE,DATA}_LEN.
This "worked for us" - YMMV.
Kjetil, how big is your cluster? Do you remember the circumstances
of when you started seeing these errors?
Andras, please let us know if this resolves the issue. Decreasing
"osd map message max" for large clusters can help with the overall
memory consumption and is probably a good idea in general, but then
these kernel client limits are pretty arbitrary, so we can look at
bumping them.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
