On 3/2/21 6:00 PM, Jeff Layton wrote:
v2 support in the kernel is keyed on the ms_mode= mount option, so that
has to be passed in if you're connecting to a v2 port. Until the mount
helpers get support for that option you'll need to specify the address
and port manually if you want to use v2.
I've tried feeding it ms_mode=v2 but I get a "mount error 22 = Invalid
argument", the ms_mode=legacy does work, but fails with the same errors.
That needs different values. See:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=00498b994113a871a556f7ff24a4cf8a00611700
You can try passing in a specific mon address and port, like:
192.168.21.22:3300:/cephfs/dir/
...and then pass in ms_mode=crc or something similar.
That said, what you're doing should be working, so this sounds like a
regression. I presume you're able to mount with earlier kernels? What's
the latest kernel version that you have that works?
5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26 Feb 2021 18:26:41
+0000 x86_64 GNU/Linux) with a cluster that has ms_bind_ipv4=false
works. Port 3300 ms_mode=prefer-crc and ms_mode=crc work.
I have tested with 5.11 kernel (5.11.2-arch1-1 #1 SMP PREEMPT Fri, 26
Feb 2021 18:26:41 +0000 x86_64 GNU/Linux) port 3300 and ms_mode=crc as
well as ms_mode=prefer-crc and that works when cluster is running with
ms_bind_ipv4=false. So the "fix" is to have this config option set: ceph
config set global ms_bind_ipv4 false
5.10 kernel (5.10.19-1-lts Arch Linux) works with a cluster that is IPv6
only but has ms_bind_ipv4=true. So it's "broken" since 5.11.
So, we have done more reading / researching on the ms_bind_ip{4,6} options:
-
https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus#Restart_the_OSD_daemon_on_all_nodes
- https://github.com/rook/rook/issues/6266
^^ Describe that you have to disable bind to IPv4.
- https://github.com/ceph/ceph/pull/13317
^^ this PR is not completely correct:
**Note:** You may use IPv6 addresses instead of IPv4 addresses, but
you must set ``ms bind ipv6`` to ``true``.
^^ That is not enough as we have learned, and starts to give trouble
with 5.11 linux cephfs client.
And from this documentation:
https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4-ipv6-dual-stack-mode
we learned that dual stack is not possible for any current stable
release, but might be possible with latest code. So the takeaway is that
the linux kernel client needs fixing to be able to support dual stack
clusters in the future (multiple v1 / v2 address families), and, that
until then you should run with ms_bind_ipv4=false for IPv6 only clusters.
I'll make a PR to clear up the documenation. Do you want me to create a
tracker for the kernel client? I will happily test your changes.
Thanks,
Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
