On 3/2/21 6:54 PM, Ilya Dryomov wrote:
--- snip ---
osd.0 up in weight 1 up_from 98071 up_thru 98719 down_at 98068
last_clean_interval [96047,98067)
[v2:[2001:7b8:80:1:0:1:2:1]:6848/505534,v1:[2001:7b8:80:1:0:1:2:1]:6854/505534,v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534]
Where did "v2:0.0.0.0:6860/505534,v1:0.0.0.0:6866/505534" come from?
This is what confuses the kernel client: it sees two addresses of
the same type and doesn't know which one to pick. Instead of blindly
picking the first one (or some other dubious heuristic) it just denies
the osdmap.
[mds.mds1{0:229930080} state up:active seq 144042 addr
[v2:[2001:7b8:80:1:0:1:3:1]:6800/2234186180,v1:[2001:7b8:80:1:0:1:3:1]:6801/2234186180,v2:0.0.0.0:6802/2234186180,v1:0.0.0.0:6803/2234186180]]
Same for the mdsmap.
Were you using ipv6 with the kernel client before upgrading to 5.11?
Yes, exclusively. So dual stack was not something that was possible
before nautilus. We always did set "ms_bind_ipv6=true".
But good find, I missed that "0.0.0.0" part. Yeah, that must be it.
What is output of "ceph daemon osd.0 config get ms_bind_ipv4" on the
osd0 node?
ceph daemon osd.0 config get ms_bind_ipv4
{
"ms_bind_ipv4": "true"
}
And
ceph daemon mds.mds1 config get ms_bind_ipv4
{
"ms_bind_ipv4": "true"
}
for that matter.
Now I'm typing this I'm like, hmm, there was an issue with this on the
mailinglist related to this, as in where you would need to disable ipv4
explicitly. I believe it was a peering issue between PGs.
OK, so this is probably it. And up to kernel 5.11 (I'll test 5.10 as
well) this would not be a problem.
I should be able te reproduce on a couple of test clusters that are
configured similarly, and be able to test if setting ms_bind_ipv4=false
on OSDs / MDSs fixes the issue.
I would like to have the heuristic to prefer IPv6 over IPv4 when
filtering addresses (as that is default / common behavior for most if
not all dual stack systems) ;-).
I'll do testing and let you know.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
