Hi,
to be more precise, netstat table looks as in the following snippet:
tcp 0 0 10.10.200.5:6815 10.10.25.4:43788 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.15.2:41020 10.10.200.8:6813 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.15.2:48724 10.10.200.9:6809 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.200.5:6813 10.10.25.4:34692 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.25.2:36736 10.10.200.6:6814 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.15.2:55250 10.10.200.8:6815 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.200.5:6815 10.10.25.6:50778 ESTABLISHED
51981/ceph-osd
tcp 0 0 10.10.15.2:57880 10.10.200.7:6803 ESTABLISHED
51981/ceph-osd
from where you can see, that source IP for all connections is always in
10.10.[12]5.x networks, which are interface addresses.
Network configuration of node is:
int vlo
ip address 10.10.200.5/32
int enp97s0f1
ip address 10.10.15.2/24
int enp19s0f0
ip address 10.10.25.2/24
where interface vlo is OVS bridge:
Bridge vlo
Port "vnet0"
Interface "vnet0"
Port vlo
Interface vlo
type: internal
and routing is similar on all nodes and looks as there
10.10.200.x proto bird metric 64
nexthop via 10.10.15.x dev enp97s0f1 weight 1
nexthop via 10.10.25.x dev enp19s0f0 weight 1
where x is address of another node with similar configuration.
As per your question, 'ceph daemon osd.0 config show' gives the following:
"cluster_addr": "10.10.200.5:0/0",
"cluster_network": "",
"cluster_network_interface": "vlo",
"public_addr": "10.10.200.5:0/0",
"public_bind_addr": "-",
"public_network": "",
"public_network_interface": "vlo"
Tomorrow we'll check source file as per your suggestion, but may be
you'll find something wrong in this config.
Thank you.
On 1/20/19 10:54 PM, Max Krasilnikov wrote:
День добрий!
Fri, Jan 18, 2019 at 11:02:51PM +0000, robbat2 wrote:
On Fri, Jan 18, 2019 at 12:21:07PM +0000, Max Krasilnikov wrote:
Dear colleagues,
we build L3 topology for use with CEPH, which is based on OSPF routing
between Loopbacks, in order to get reliable and ECMPed topology, like this:
...
CEPH configured in the way
You have a minor misconfiguration, but I've had trouble with the address
picking logic before, on a L3 routed ECMP BGP topography on IPv6 (using
the Cumulus magic link-local IPv6 BGP)
[global]
public_network = 10.10.200.0/24
Keep this, but see below.
[osd.0]
public bind addr = 10.10.200.5
public_bind_addr is only used by mons.
cluster bind addr = 10.10.200.5
There is no such option as 'cluster_bind_addr'; it's just 'cluster_addr'
Set the following in the OSD block:
| public_network = # keep empty; empty != unset
| cluster_network = # keep empty; empty != unset
| cluster_addr = 10.10.200.5
| public_addr = 10.10.200.5
Alternatively, see the code src/common/pick_address.cc to see about
using cluster_network_interface and public_network_interface.
Unfortunatelly, all osds continue to bind to interface addresses instead of vlo
bridge address even after setting cluster_addr, public_addr,
cluster_network_interface and public_network_interface :(
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
