Digging further and checking osd metadata I have what it seems like a bug
assigning the addresses.
In most OSDs looking at ceph osd metadata the result is fine and the front
addresses are correctly configured through the 10.113 public network, like
this one:
== osd.0 ==
"back_addr": "[v2:
10.114.29.10:6813/2947358317,v1:10.114.29.10:6819/2947358317]",
"front_addr": "[v2:
10.113.29.10:6801/2947358317,v1:10.113.29.10:6807/2947358317]",
"hb_back_addr": "[v2:
10.114.29.10:6837/2947358317,v1:10.114.29.10:6843/2947358317]",
"hb_front_addr": "[v2:
10.113.29.10:6825/2947358317,v1:10.113.29.10:6832/2947358317]",
But then, there are also several cases where the configuration is
incorrect, but this could happen in different ways.
For example in some OSDs the error is just in the front_addr, but the
hb_front_addr is fine:
== osd.26 ==
"back_addr": "[v2:
10.114.29.5:6866/4155549673,v1:10.114.29.5:6867/4155549673]",
"front_addr": "[v2:*10.114.29.5*:6864/4155549673,v1:
10.114.29.5:6865/4155549673]",
"hb_back_addr": "[v2:
10.114.29.5:6870/4155549673,v1:10.114.29.5:6871/4155549673]",
"hb_front_addr": "[v2:
10.113.29.5:6868/4155549673,v1:10.113.29.5:6869/4155549673]",
In others it is the hb_front_addr:
== osd.34 ==
"back_addr": "[v2:
10.114.29.6:6802/3934363792,v1:10.114.29.6:6803/3934363792]",
"front_addr": "[v2:
10.113.29.6:6800/3934363792,v1:10.113.29.6:6801/3934363792]",
"hb_back_addr": "[v2:
10.114.29.6:6806/3934363792,v1:10.114.29.6:6807/3934363792]",
"hb_front_addr": "[v2:*10.114.29.6*:6804/3934363792,v1:
10.114.29.6:6805/3934363792]",
And in others both are wrong:
== osd.32 ==
"back_addr": "[v2:
10.114.29.10:6814/2403531529,v1:10.114.29.10:6820/2403531529]",
"front_addr": "[v2:*10.114.29.10*:6802/2403531529,v1:
10.114.29.10:6808/2403531529]",
"hb_back_addr": "[v2:
10.114.29.10:6836/2403531529,v1:10.114.29.10:6841/2403531529]",
"hb_front_addr": "[v2:*10.114.29.10*:6826/2403531529,v1:
10.114.29.10:6830/2403531529]",
This happens just for the front_addr assignation, the back_addr in all OSDs
is in the cluster network (10.114).
The ceph cluster was upgraded and rebooted so I discard a stalled
configuration.
I will file a bug just in case.
On Wed, 29 Sept 2021 at 10:53, Javier Cacheiro <Javier.Cacheiro@xxxxxxxxx>
wrote:
> Hi David,
>
> After some investigation, it seems that it affects all osd nodes, in a way
> that the same node sometimes is being announced through the cluster_network
> (10.114) and others through the public_network (10.113).
>
> Since each node has 12 OSDs I suspect it could depends on the specific
> OSD. This seems to discard a network misconfiguration because OSDs in the
> same node are being announced through cluster or public network.
>
> I will look for further information to verify this, but now I am inclined
> to think that this could be a bug introduced in 16.2.6.
>
> >> for n in {1..100}; do strace -f -e trace=network -s 10000 rbd info
> benchimage-test-c27-35-$n --pool glance-images --name client.glance 2>&1|
> grep sin_addr | egrep '(10.113.29|10.114.29)'| awk '{print $6}'; done |
> sort | uniq -c
> 15 sin_addr=inet_addr("10.113.29.1")},
> 18 sin_addr=inet_addr("10.113.29.10")},
> 29 sin_addr=inet_addr("10.113.29.11")},
> 12 sin_addr=inet_addr("10.113.29.12")},
> 45 sin_addr=inet_addr("10.113.29.13")},
> 27 sin_addr=inet_addr("10.113.29.14")},
> 310 sin_addr=inet_addr("10.113.29.15")},
> 29 sin_addr=inet_addr("10.113.29.16")},
> 67 sin_addr=inet_addr("10.113.29.2")},
> 15 sin_addr=inet_addr("10.113.29.3")},
> 63 sin_addr=inet_addr("10.113.29.4")},
> 24 sin_addr=inet_addr("10.113.29.5")},
> 18 sin_addr=inet_addr("10.113.29.6")},
> 24 sin_addr=inet_addr("10.113.29.7")},
> 20 sin_addr=inet_addr("10.113.29.8")},
> 39 sin_addr=inet_addr("10.113.29.9")},
> 9 sin_addr=inet_addr("10.114.29.1")},
> 3 sin_addr=inet_addr("10.114.29.10")},
> 3 sin_addr=inet_addr("10.114.29.14")},
> 294 sin_addr=inet_addr("10.114.29.2")},
> 21 sin_addr=inet_addr("10.114.29.3")},
> 3 sin_addr=inet_addr("10.114.29.4")},
> 33 sin_addr=inet_addr("10.114.29.5")},
> 25 sin_addr=inet_addr("10.114.29.8")},
>
>
> On Tue, 28 Sept 2021 at 18:15, David Caro <dcaro@xxxxxxxxxxxxx> wrote:
>
>>
>> Just curious, does it always happen with the same OSDs?
>>
>> On 09/28 16:14, Javier Cacheiro wrote:
>> > Interestingly enough this happens for some pools and not for others.
>> >
>> > For example I have just realized that when trying to connect to another
>> > pool the client is correctly directed to the OSD public_network address:
>> >
>> > >> strace -f -e trace=network -s 10000 rbd ls --pool cinder-volumes
>> --name
>> > client.cinder 2>&1| grep sin_addr
>> > [pid 2363212] connect(15, {sa_family=AF_INET, sin_port=htons(6816),
>> > sin_addr=inet_addr("*10.113.29.7*")}, 16) = 0
>> >
>> > But same client listing the ephemeral-vms pools is directed to the OSD
>> > cluster address:
>> > >> strace -f -e trace=network -s 10000 rbd ls --pool ephemeral-vms
>> --name
>> > client.cinder 2>&1| grep sin_addr
>> > [pid 2363485] connect(14, {sa_family=AF_INET, sin_port=htons(6806),
>> > sin_addr=inet_addr("*10.114.29.10*")}, 16) = -1 EINPROGRESS (Operation
>> now
>> > in progress)
>> >
>> > Very weird!
>> >
>> >
>> >
>> > On Tue, 28 Sept 2021 at 16:02, Javier Cacheiro <
>> Javier.Cacheiro@xxxxxxxxx>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I am trying to understand a issue with ceph directing clients to
>> connect
>> > > to OSDs through their cluster_network address instead of their
>> > > public_network address.
>> > >
>> > > I have a configured a ceph cluster with a public and cluster network:
>> > >
>> > > >> ceph config dump|grep network
>> > > global advanced cluster_network *10.114.0.0/16
>> > > <http://10.114.0.0/16>* *
>> > > mon advanced public_network 10.113.0.0/16 *
>> > >
>> > > I upgraded the cluster from 16.2.4 to 16.2.6.
>> > >
>> > > After that, I am seeing that ceph is directing clients to connect to
>> OSD's
>> > > cluster_network address instead of their public_address:
>> > >
>> > > >> strace -f -e trace=network -s 10000 rbd ls --pool ephemeral-vms
>> --name
>> > > client.cinder
>> > > ....
>> > > [pid 2353692] connect(14, {sa_family=AF_INET, sin_port=htons(6806),
>> > > sin_addr=inet_addr("*10.114.29.10*")}, 16) = -1 EINPROGRESS (Operation
>> > > now in progress)
>> > >
>> > > In this case the client hangs because it is not able to access the
>> > > address, since its an internal address.
>> > >
>> > > This appeared after upgrading to 16.2.6, but I am not sure it was due
>> to
>> > > the upgrade or it was a hidden issue that appeared after the nodes
>> were
>> > > rebooted.
>> > >
>> > > It can also be that I am missing something in the config, but this
>> config
>> > > was generated by the cephadm bootstrap command and not created by
>> hand, and
>> > > it worked before the upgrade/reboot so I am pretty confident with it.
>> > >
>> > > What do you think, can this be a bug or is more a misconfiguration on
>> my
>> > > side?
>> > >
>> > > Thanks,
>> > > Javier
>> > >
>> > >
>> > >
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>> --
>> David Caro
>> SRE - Cloud Services
>> Wikimedia Foundation <https://wikimediafoundation.org/>
>> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
>>
>> "Imagine a world in which every single human being can freely share in the
>> sum of all knowledge. That's our commitment."
>>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
