On 3/2/21 5:16 PM, Jeff Layton wrote:
On Tue, 2021-03-02 at 09:25 +0100, Stefan Kooman wrote:
Hi,
On a CentOS 7 VM with mainline kernel (5.11.2-1.el7.elrepo.x86_64 #1 SMP
Fri Feb 26 11:54:18 EST 2021 x86_64 x86_64 x86_64 GNU/Linux) and with
I'm guessing this is a stable series kernel
It's a kernel from 'elrepo', so I'm not sure. I would guess so too.
Ceph Octopus 15.2.9 packages installed. The MDS server is running
Nautilus 14.2.16. Messenger v2 has been enabled. Poort 3300 of the
monitors is reachable from the client. At mount time we get the following:
Mar 2 09:01:14 kernel: Key type ceph registered
Mar 2 09:01:14 kernel: libceph: loaded (mon/osd proto 15/24)
Mar 2 09:01:14 kernel: FS-Cache: Netfs 'ceph' registered for caching
Mar 2 09:01:14 kernel: ceph: loaded (mds proto 32)
Mar 2 09:01:14 kernel: libceph: mon4 (1)[mond addr]:6789 session established
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec
Mar 2 09:01:14 kernel: ceph: corrupt mdsmap
Mar 2 09:01:14 kernel: ceph: error decoding mdsmap -22
-22 == -EINVAL
Looks like a an osdmap parsing error?
Indeed, that is kinda weird isn't it.
Mar 2 09:01:14 kernel: libceph: another match of type 1 in addrvec
Mar 2 09:01:14 kernel: libceph: corrupt full osdmap (-22) epoch 98764 off 6357 (0000000027a57a75 of 00000000d3075952-00000000e307797f)
Mar 2 09:02:15 kernel: ceph: No mds server is up or the cluster is laggy
The /etc/ceph/ceph.conf has been adjusted to reflect the messenger v2
changes. ms_bind_ipv6=trie, ms_bind_ipv4=false. The kernel client still
seems to be use the v1 port though (although since 5.11 v2 should be
supported).
The mount helper only recently got v2 support, and that hasn't trickled
out into the distros yet. See:
https://github.com/ceph/ceph/pull/38788
Ah, good to know.
Has anyone seen this before? Just guessing here, but could it that the
client tries to speak v2 protocol on v1 port?
What mount options are you passing in? Are you using mon autodiscovery?
We provide the mon address explictly in the /etc/fstab, So something
like this:
mon1,mon2,mon3,mon4,mon5:/cephfs/dir /client_mountpoint ceph
name=client-id,secretfile=/etc/ceph/ceph.client.client-id.cephfs.key,noatime,_netdev
0 2
We are not using any dns based discovery of monitors if that is what you
mean.
Note: We tried with nautilus packages before (14.2.16) and got the same
result.
v2 support in the kernel is keyed on the ms_mode= mount option, so that
has to be passed in if you're connecting to a v2 port. Until the mount
helpers get support for that option you'll need to specify the address
and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid
argument", the ms_mode=legacy does work, but fails with the same errors.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
